Extensible model loading using a more modular resource mechanism

[coldfix]

Hey,

I felt the need to have PyMad load models from more general sources, such as a custom python package. This is why I modified, once again, the model loading mechanism. This time the modification goes much deeper but is also designed a lot better. The new model loading works as follows:

• resources (files) are loaded via ResourceProvider objects. I have implemented ResourceProviders for couchdb, python package and normal filesystem resources.
• models are located using a ModelLocator object, that can lookup models accessible via a ResourceProvider. This mechanism might be merged with the existing cpymad.service module, but I am not fully sure about this. For now I have modified cpymad.service to use the ModelLocator.

Before this is ready to merge, there are still some open issues:

The couchdb code probably doesn't work yet, as I have no models in a couchdb to test with. On the other hand, from what I saw in the existing code, the couchdb wasn't functional before my modifications either. However, with the new resource mechanism, it should be easier to get it to work.

In the course of the week, I will also modify some docstrings to be more sphinx-friendly.

On my machine, except for test_lhc-b4 and test_survey all of the existing tests in the test/cpymadtest folder are failing with serious runtime errors (such as memory faults) within the Mad-X library. This was already the case on your last master version. One reason might be, that I have no AFS installed at the moment. I haven't checked if there is any version the tests succeed on my machine. Nevertheless, this means, that I cannot verify that the changes in this pull request do not break anything.

Do these tests also fail on your setup? If so, this might be an important issue to fix. If not, I could send you error traces, as I have currently no clue why these errors emerge.

By the way, I have also started a python3 compatibility branch, for which it might be especially important to have the tests fixed before merging (as python3 features incompatible behaviour regarding unicode and binary IO).

If something is unclear, I will be glad to explain it in further detail.

I have placed the cern/resource/* files under a quasi-public domain (CC0) license, such that it is easier to extract them from the project if they prove useful in a different location. For some of the other files, license notices are still missing, I will try to add them in the next days. Should I copy the license notices from other files in the tree as is, meaning the license owner will be CERN? I am not sure whether I am allowed to that. On the other hand I don't know if it makes the "overall" license more complicated if I enter my own name. As far as I'm concerned, I don't care.

Sincerely,

Thomas

[Eothred]

Wow, this sounds very nice, thanks!

For the tests, a basic principle of the pymad code is that it is supposed to run without the need for afs. However as my computer usually have afs available, I often don't notice that the tests are failing without afs. I will investigate that bit. I should also mention that I have runtime errors with the libmadx when compiled with e.g. intel compilers, which I think happens because python is more strict when libraries tries to access memory out of range (?).

For couchdb you are right. I think I did some basic tests when we had a server available back in the days, but I don't think I ever finished that feature. The idea was that since we use json files, and couchdb essentially is a server based on the json format, this would be a convenient way to store models in a common location (not locked into pymad).

For docstrings I should apologise for being so late updating our webpages. I did so a few days ago, so your new documentation for matching etc is now available.

License.. Hmm, good question. I don't think they have any at the moment, which I guess puts them under "all rights reserved". No one wants to care about licenses, but they can cause serious problems if not taken into account. I will investigate that bit. The files on afs are meant to be used by anyone for any purpose, so I am sure the most correct license to put them under would be some very relaxed CC license.

It will take me some time to go through this merge request.

[coldfix]

I some more remarks:

• I have deleted the listModels.py file, which is not necessary anymore inside pymad. If this breaks any code, I could add a simple replacement
• 'extends' for model definitions now works somewhat different than before. First: a deep merge is performed, not just on a single level. Second: the mechanism is transitive (bases of bases are considered). Third: properties in 'derived' models overwrite properties of bases. I thought this to be the more correct approach. I hope, it does not break something.
• Which python version(s) are you targeting? In my experience, supporting anything lower than 2.6 is really difficult. I guess everybody should have 2.7 by now.

[Eothred]

Target Python 2.6 and 2.7 for now, you got that right :) Python 3 would be nice in the future I guess, but even now I don't know many people who use it (at least as main version). 2.5 and below is too much hassle, and I even Scientific Linux CERN now have 2.6.

Your definition of extends and derived sounds correct. I added the extends bit rather quickly because I got tired of redefining the same thing over and over again. Only the 'lhc' model file is using this for now, and this file still works during testing.

I think you are right about the listModels file. I will check just to make sure (there is a command in the common abi to list all available models).

For the licenses, I have no idea which is better, but I was thinking maybe it would be better to use one license for the entire project (if you would accept). We used the Apache 2 license for the other parts simply because it is very free and allows any use in the future including commercial. The CC0 license you used sounds similar to me (but I am no lawyer and haven't looked deeply into this matter).

I had a look at which tests are passing for me. There are some which are failing, the ILC models and the lhc_lsa test. I was thinking to add ctest/cdash test files so we can run the tests regularly from a server and keep track of this. See the ctest_cdash branch and http://cern.ch/abp-cdash/index.php?project=pymad&date=2013-10-09 Do you know of better dashboard solutions for this kind of stuff? I simply use cdash because I already know cmake from other projects and find it reasonably convenient.. And also we have the server already available.

Cheers, Yngve

[coldfix]

I agree it is probably be the easiest to have everything under the same license. Having some files public domain shouldn't be a problem, though, since public domain basically means you can do everything with them. If you don't want to keep the additional license file, I'd publish these somewhere else and then import them into pymad. What I don't like very much about the apache license is that you can't relicense the source and you have to explicitly mark all modifications from the original. I'm not even sure whether a git commit history is sufficient for that purpose.

Regarding automated testing: I have only worked with the Travis continiuous integration service so far. It is a pretty nice tool for testing on multiple python versions and it fetches and tests every commit pushed on github automatically. Problematic would be repeated recompilation of the madx-source. It would probably be the best to have a binary distribution of madx.

[Eothred]

Regarding license: Essentially, if you selected a more relaxed license than what we use (and compatible) I have really no issue with that of course. I'm just not a lawyer ;)

Travis looks convenient yes, but I don't see right now how I would instruct it to e.g. compile the shared Mad-X libraries. I'll take it under consideration, thanks.

I think I will merge this request soon, I don't see any obvious problems with it at least (and I welcome the help/additions!)

[Eothred]

I added you in the new "Developer" team, since you have proven to produce nice stuff over time now. You should be able to directly push to the main repository.

Thanks again!