airavata-courses / TeamCodeRing

0 stars 0 forks source link

Remaining Issues #39

Open marpierc opened 7 years ago

marpierc commented 7 years ago

You still need to address these from the PM4 assignment in canvas. My comments are in italics.

Nethracs commented 7 years ago
  1. we are using mesos to execute a spark task. note that spark task is triggered from data ingestor. You can see the details of the same in the satellite app (http://ec2-35-166-177-93.us-west 2.compute.amazonaws.com/satellite) which is the registry. So a spark submit command triggers a procedure in mesos whose logs are available in the data ingestor app. We append the details of this in the registry app.

  2. The job progress data is displayed in a well formatted manner and it is diaplayed at http://ec2-35-166-177-93.us-west-2.compute.amazonaws.com/satellite

  3. Submitted requests in http://ec2-35-166-177-93.us-west-2.compute.amazonaws.com/ allows user to view all previously submitted jobs by user.

  4. Modify previously submitted job displays previously submitted jobs progress and populates the station field with data. Could not populate date field due to the issues with date time picker format.

  5. The data ingestor and spark microservice is the one which uses mesos, it triggers the execution with a spark submit and results will be updated to redis which can viewed through satellite. We have all the nodes running this microservice so a failure of one or few nodes will not stop behaviour.

Nethracs commented 7 years ago

Please see the mesos trigger from data ingestor at https://github.com/airavata-courses/TeamCodeRing/blob/Dockerization/rain_gauge/application.py