search

cvmfs-contrib / config-repo

Master source for CVMFS configuration repository for major organizations
4 stars 20 forks source link

Adding the storage.igwn.org domain #160

Closed djw8605 closed 1 year ago

djw8605 commented 1 year ago

My initial attempt at adding the storage.igwn.org domain. virgo.storage.igwn.org repo is ready to go, it's on http://hcc-cvmfs-repo.unl.edu:8000/cvmfs/virgo.storage.igwn.org (firewalled to only stratum 1s)

The config file is largely adapted from the ligo.osgstorage.org repo. Do I need to add some lines from the osgstorage.org.conf from domains.d ?

djw8605 commented 1 year ago

Note that this is an authenticated CVMFS. So we can’t copy the external URL from osgstorage.org

DrDaveD commented 1 year ago

Right, so we additionally want all the settings in ligo.osgstorage.org.conf, which will override those external URL settings but not override everything. For that we probably ought to have a common config file, for example /etc/cvmfs/authosgstorage.conf and have config.d/ligo.osgstorage.org.conf and config.d/ligo.storage.igwn.org.conf and config.d/virgo.storage.igwn.org.conf symlink to it.

djw8605 commented 1 year ago

Ah, so it gets more complicated.

The ligo.osgstorage.org external URLs all have the path /user/ligo appended. The new scheme will not any path appended. Therefore, ligo.osgstorage.org and virgo.storage.igwn.org do not have any shared external urls.

But, we could have a shared authenticated osg common config, though there wouldn't be a common domain, so I suppose we would have to just "import" it with a . ? The common osg config will be shared by... well I guess just all the domains under storage.igwn.org. So maybe, for now, just have a singular storage.igwn.org config?

DrDaveD commented 1 year ago

Oh that was quite a bad choice. I'm glad it is being corrected. However it is very unfortunate to have to put the same list of servers in two different files. I suppose we can assume that the ligo.osgstorage.org repository will be going away, is that your expectation? If so maybe this is only a temporary problem.

Other than the list of external URLs, everything configured in ligo.osgstorage.org.conf is already in another file, libexec/osgstorage-auth.conf.

In spite of this complication, I still think that it's better to not put authenticated config into storage.igwn.org.conf, because someday they may want an unauthenticated external data repo. Also that allows us to use symlinks instead of "." which is simpler because we don't need to define all the exported variables.

So, assuming we stick with a separate config file for authenticated osgstorage, I would still go ahead and update ligo.osgstorage.org.conf to "." it, but then use shell substitutions to append /user/ligo to all the servers. Probably we should move libexec/osgstorage-auth.conf to etc/cvmfs/osgstorage-auth.conf and add in the CVMFS_EXTERNAL_URL settings to it without /user/ligo in the server URLs.

djw8605 commented 1 year ago

Right, ligo.osgstorage.org is going away. So I would rather not make any major or any changes to the ligo config, let it die slowly.

I'll see if I can make the changes correctly to move the osgstorage-auth conf over.

djw8605 commented 1 year ago

@DrDaveD ok, I think I made the necessary changes. Right now, I just have ligo overwriting the external url after importing the osgstorage-auth.conf file. My bash-foo isn't strong enough to the shell replacement without likely a few hours of refresher.

DrDaveD commented 1 year ago

I added a commit that arranges things the way I had in mind. Please confirm that it looks OK to you.

I also split up the CVMFS_EXTERNAL_URL URLs each onto their own line, something we had been talking about for a while for easier maintenance.

djw8605 commented 1 year ago

All looks good to me.