search

galaxyproject / idc

Simon's Data Club - Reference data for Galaxy servers
MIT License
9 stars 7 forks source link

DM loading in the build instance is a hack #35

Open natefoo opened 8 months ago

natefoo commented 8 months ago

On the builder instance database_connection is Main's DB since we want DM builds to be persistent on Main. This means that config watchers are also Main's config watchers, and so if we install DMs on the build instance, the install process never completes because Main's config watcher doesn't see the change to shed_data_managers_conf.xml.

As a workaround, we preinstall DMs properly on Main in galaxyproject/usegalaxy-tools. Ideally we could just load /cvmfs/main.galaxyproject.org/config/shed_data_manager_conf.xml, but this not possible since the shed_tool_conf.xml referenced in the shed_data_manager_conf.xml has to be loaded:

[pN:idc-build,p:19389,tN:MainThread] '/cvmfs/main.galaxyproject.org/config/shed_tool_conf.xml' not among installable tool config files (/srv/galaxy/var/config/shed_tool_conf.xml, /srv/galaxy/var/config/migrated_tools_conf.xml)

As a result, we manually add to and deploy a non-shed data_manager_conf.xml in the playbook. This is an extra step any time you need to install or update a DM, and also has the downside of making our DMs unversionable (we install shed versioned DMs but load them in an "local" unversioned way).

The proper solution is probably a Galaxy fix to make it possible to load shed_data_manager_conf.xml without the corresponding shed_tool_conf.xml.

xref full discussion on matrix

mvdbeek commented 8 months ago

The proper solution is probably a Galaxy fix to make it possible to load shed_data_manager_conf.xml without the corresponding shed_tool_conf.xml.

I think I would instead decouple the config watching election and messaging from the database. Or as a temporary hack it'd be sufficient to name the idc instance zzz which would win the config watching election (which is reverse sorting by process name ...)

natefoo commented 8 months ago

name the idc instance zzz

Hacks upon hacks!