search

openedx-unsupported / edx-analytics-configuration

GNU Affero General Public License v3.0
8 stars 28 forks source link

Script to run shell command on remote cluster #73

Closed HassanJaveed84 closed 5 years ago

HassanJaveed84 commented 5 years ago

I just thought it was simpler as the remote-task already implemented a shell command calling feature. All we really need to do is to call s3-dist-cp on the master node, this could either be done by executing a shell command, adding a step to the EMR cluster, or like you said wrapping it in a luigi task. Also, with the Luigi task, it would go through all those steps running ansible playbook to the cluster etc.

Regarding the logging, everything just goes to the console. Do we need to change that ?

brianhw commented 5 years ago

Nope. I approved the PR already. I was just curious. Luigi runs lots of scripts, so I was just wondering if there was a problem you ran into that would be good to know about, or if it were just a different alternative. Sounds like the latter.