thasso / pyjip

JIP Pipeline System
http://pyjip.readthedocs.org
Other
19 stars 8 forks source link

Develop #54

Closed Poshi closed 10 years ago

Poshi commented 10 years ago

Main improvement is the addition of MySQL support and lowering DB requirements.

thasso commented 10 years ago

Hey @Poshi and @emi80 , I merged all changes except 2f92f9a (the mysql feature) cause @emi80 did something very similar on the develop branch. I get conflicts in the db setup, most are trivial, but two are on job deletion and table creation. Maybe somebody can take a look ?

Poshi commented 10 years ago

I'm feeling like an idiot. There is a way to see the conflicts? I only manage to see the differences, but not the conflicts, so I cannot tell anything about them :-(

thasso commented 10 years ago

Nope, unfortunately I also don't know how to see the commits here :( I just added you repo/branch as described in "Use the command line to resolve conflicts" and then I cherry picked the commits, and there I saw the conflicts of course.

Poshi commented 10 years ago

I re-cloned your repository and I'm testing the @emi80 changes. It connects to the MySQL database, but it does not solve one important problem: each jip-exec needs two open connections to the DB for the duration of the execution, which consumes DB resources for nothing. This connection is only used at the beginning and at the end to set the statuses, but in the middle of the execution is useless. I will try to re-add the code needed to solve this issues and issue another pull request, hopefully without merge conflicts.

Poshi commented 10 years ago

OK, I have the code ready and tested in our cluster. On Monday I will prepare a git pull request, because I'm not able to push to your repo directly.

thasso commented 10 years ago

Cool! Thanks Jordi. BTW, I just added you to the commiter list, so you should be abel to push directly and/or accept pull requests!

Poshi commented 10 years ago

OK, I just commited the changes. Maybe @emi80 can test it in its pipelines. BTW, I'm going to cancel this pull request, it is no longer necessary.

emi80 commented 10 years ago

Hi guys, sorry for my late reply. I was on vacation. Thanks @Poshi for the work on MySQL support. I will test it on a real pipeline asap. @thasso, are you planning a new release with these changes?

thasso commented 10 years ago

Also sorry for my side for the delay. Yeah, if you guys can confirm that its working okay, I can do the release. No problem.

2014-08-25 14:24 GMT+02:00 Emilio notifications@github.com:

Hi guys, sorry for my late reply. I was on vacation. Thanks @Poshi https://github.com/Poshi for the work on MySQL support. I will test it on a real pipeline asap. @thasso https://github.com/thasso, are you planning a new release with these changes?

— Reply to this email directly or view it on GitHub https://github.com/thasso/pyjip/pull/54#issuecomment-53257152.

Poshi commented 10 years ago

Hey! This time it was me that was on vacation. I've not done extensive testing, but "toy testing" seems to be ok. BTW, I have the code to add a pipeline name to the jobs (not the programmatic name, but a user defined name to identify the pipeline amongs other similar ones). Just testing it now. I can push this changes too.

Jordi Camps jcamps@gmail.com

-----BEGIN GEEK CODE BLOCK----- Version: 3.12 GCS/IT/S d-(+) s: a C++ UL++++() P+ L+(++) E- W++ N o? K? w++ !O M V PS++ PE- Y? PGP t 5 X R(+) tv(+) b+ DI+ D+ G e+++ h r(--) y+** ------END GEEK CODE BLOCK------

2014-09-04 9:28 GMT+02:00 thasso notifications@github.com:

Also sorry for my side for the delay. Yeah, if you guys can confirm that its working okay, I can do the release. No problem.

2014-08-25 14:24 GMT+02:00 Emilio notifications@github.com:

Hi guys, sorry for my late reply. I was on vacation. Thanks @Poshi https://github.com/Poshi for the work on MySQL support. I will test it on a real pipeline asap. @thasso https://github.com/thasso, are you planning a new release with these changes?

— Reply to this email directly or view it on GitHub https://github.com/thasso/pyjip/pull/54#issuecomment-53257152.

— Reply to this email directly or view it on GitHub https://github.com/thasso/pyjip/pull/54#issuecomment-54421830.

thasso commented 10 years ago

Sonds okey to me. My suggestion would be, you push your changes for the pipeline name, we update documentation and maybe beginning of next week we release ?

Poshi commented 10 years ago

Changes pushed. The changes should not break anything besides the own pipeline name management. About documentation... I think there is no place in the docs to talk about this.

Jordi Camps jcamps@gmail.com

-----BEGIN GEEK CODE BLOCK----- Version: 3.12 GCS/IT/S d-(+) s: a C++ UL++++() P+ L+(++) E- W++ N o? K? w++ !O M V PS++ PE- Y? PGP t 5 X R(+) tv(+) b+ DI+ D+ G e+++ h r(--) y+** ------END GEEK CODE BLOCK------

2014-09-19 12:43 GMT+02:00 thasso notifications@github.com:

Sonds okey to me. My suggestion would be, you push your changes for the pipeline name, we update documentation and maybe beginning of next week we release ?

— Reply to this email directly or view it on GitHub https://github.com/thasso/pyjip/pull/54#issuecomment-56162183.