Closed keenanlang closed 5 years ago
keenanlang
commented
5 years ago @anjohnson, are there any major things that stand out as a potential issue in how a synApps module might be set up that could cause this automatic dependency generation to have issues? Or suggestions on how to accomplish this better?
timmmooney
commented
5 years ago There is one weird little twist that you probably already know about. Just in case:
The CALC module can build with or without the SSCAN module, and with or without the SNCSEQ module. Any module with a build-time dependency on CALC must define SSCAN in its RELEASE file if CALC defines SSCAN, and must define SNCSEQ if CALC defines SNCSEQ.
The SNCSEQ dependency is just a bit of fluff (editSseq.st) which I think nobody but me may ever have used. The SSCAN dependency is more substantial: CALC builds the swait record if it can depend on SSCAN, because the swait record uses recDynLink, which is defined in SSCAN. The swait record doesn't actually need anything in CALC; it uses the calc engine in base.
Tim Mooney (mooney@anl.gov) (630)252-5417 Beamline Controls Group (www.aps.anl.gov) Advanced Photon Source, Argonne National Lab
From: Keenan Lang notifications@github.com Sent: Tuesday, April 30, 2019 5:43 PM To: EPICS-synApps/support Cc: Subscribed Subject: [EPICS-synApps/support] Modular dependency (#11)
Change Makefile to build the graph of interdependencies between modules automatically from their RELEASE files. Relies on a new script depends.pl that has been added to utils (https://github.com/EPICS-synApps/utils/blob/master/depends.pl) which, in turn, uses convertRelease.pl from base.
Depends.pl takes in the path to convertRelease, the path to the module, the name of the module, and the list of potential modules to build against. It reads the module's RELEASE file for any macros defined, removes any macros that aren't one of the modules in MODULE_LIST and removes macros that have the same name as the module itself. What remains should be a list of the directories that the module needs to build against.
I have found one issue with this, where AreaDetector doesn't require autosave, busy, etc to build its core, but does to build its IOC's. The macros aren't defined in RELEASE, but in RELEASE_PRODS.local, which is only included by the IOC's. Though, this seems more like an issue to be solved with tweaks to AreaDetector rather than changing something here.
You can view, comment on, or merge this pull request online at:
https://github.com/EPICS-synApps/support/pull/11
Commit Summary
File Changes
Patch Links:
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/EPICS-synApps/support/pull/11, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABT6PF26FJFH4IKTKVLADA3PTDDPPANCNFSM4HJQWROQ.
keenanlang
commented
5 years ago This wouldn't end up altering anything regarding build sequence or introduce new issues regarding those kinds of optional dependencies.
If a module that includes CALC doesn't also define SSCAN or SNCSEQ in their RELEASE file, they won't have those modules in their DEPENDS_DIRS, but they still depend on CALC and CALC will depend on one or both of SSCAN/SNCSEQ, so whatever module is including CALC won't be build until the necessary modules are built and then the build will fail in the same way that it currently does.
One benefit, though, is that when CALC isn't built with either of those two modules, parallel builds will go slightly faster then they do today because we wouldn't have an explicit dependency on SSCAN and SNCSEQ.
anjohnson
commented
5 years ago As part of the Upgrade the new APS Accelerator IOCs and all their support modules will be built using the Sumo tool from BESSY. This handles dependencies automatically at build configuration time — you add a configure/MODULES file to the IOC which Sumo turns into a configure/RELEASE file for it. Sumo manages a dependency database (stored as a JSON file in a git repository) that defines for each tagged module version what dependencies it has, where the source code for it can be found (a git/hg/svn/tarfile URL plus tag) and any special make recipes needed. It also supports module aliases, so you can use the module name BASE in the database but in your configure/RELEASE file it will show up correctly as the variable name EPICS_BASE.
You can update just the tag for say the BASE version in an IOC's configure/MODULES file and Sumo will look at all the dependent modules, work out which ones are already available for the new Base version and (as long as there are no conflicts) build everything else for you automatically and in the right order. It checks out each module in the build area, creates a new configure/RELEASE file (from scratch) that points to its dependencies, and then (optionally) runs the build. If everything needed is already available though it only has to create the IOC's configure/RELEASE file pointing to those versions.
The main limitation I have seen with Sumo so far is that it doesn't seem to handle optional dependencies, so (referring to @timmmooney's comment) for the CALC module I have BASE and SSCAN listed as dependencies, but not SEQ. However each of my modules also has a registered version master which checks out the tip of the master branch, and that can have different dependencies so for development purposes I could ask Sumo to check out a local copy of the master branch, make changes and run the tests against that (with SEQ enabled) but when I commit and tag a new version I would leave off the SEQ dependency again since it's only needed for development.
Sumo does have the ability to specify version dependencies too, so this module version A needs at least version X of a dependency but only up to version Y. I haven't played with this yet though, I'm still working on getting an initial set of modules imported for my customers to start using.
Why am I telling you this? Well it's another approach which BESSY have been using for many years and seems to work pretty well. I'm not sure how/if it will cope with building AreaDetector though, I haven't tackled that yet either. I'm not suggesting you should scrap your new script or the approach you guys have developed, just keep an eye on what else is out there — I don't see any obvious problems with the approach you described but I haven't looked at it in detail.
Change Makefile to build the graph of interdependencies between modules automatically from their RELEASE files. Relies on a new script depends.pl that has been added to utils (https://github.com/EPICS-synApps/utils/blob/master/depends.pl) which, in turn, uses convertRelease.pl from base.
Depends.pl takes in the path to convertRelease, the path to the module, the name of the module, and the list of potential modules to build against. It reads the module's RELEASE file for any macros defined, removes any macros that aren't one of the modules in MODULE_LIST and removes macros that have the same name as the module itself. What remains should be a list of the directories that the module needs to build against.
I have found one issue with this, where AreaDetector doesn't require autosave, busy, etc to build its core, but does to build its IOC's. The macros aren't defined in RELEASE, but in RELEASE_PRODS.local, which is only included by the IOC's. Though, this seems more like an issue to be solved with tweaks to AreaDetector rather than changing something here.