search

dmwm / CRABServer

16 stars 38 forks source link

Run spark cronjobs on ipynb directly #8735

Closed novicecpp closed 1 month ago

novicecpp commented 1 month ago

Fix https://github.com/dmwm/CRABServer/issues/7798

Instead running on py file where it need to copy the contents from Jupyter notebook back and forth between dev/PoC to prod, we make notebook files to be able to run as cronjobs instead. To debug those notebook files in the future, you simply copy to CERN's Jupyter and run it directly as notebook. When you finish fixing code, or creating a new one, simply clean up notebook output and copy back to crab-spark/notebooks.

You need to be careful when crafting those notebooks though. But it is simply the template that you need to follow.

Note that this PR includes the fixes that work with the current setup we have with Monit team.

I will wrote it down the instruction after it merge to master tomorrow.

novicecpp commented 1 month ago

@belforte Stefano, I did not ask for a review from you this time and kinda force you to accept my approach. But, I can guarantee that you can revert this behavior back to a simple py file later if it does not work or you do not like it.

We can have some chat tomorrow in the meeting.

cmsdmwmbot commented 1 month ago

Jenkins results:

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-CRABServer-PR-test/2189/artifact/artifacts/PullRequestReport.html

cmsdmwmbot commented 1 month ago

Jenkins results:

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-CRABServer-PR-test/2191/artifact/artifacts/PullRequestReport.html

cmsdmwmbot commented 1 month ago

Jenkins results:

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-CRABServer-PR-test/2196/artifact/artifacts/PullRequestReport.html

novicecpp commented 1 month ago

This PR is running daily in dev without any issue.

I need to merge this to master first before I can put in production.