Add ardc.edu.au domain (osg)

[hzhu16]

Hi, this is a new PR for the osg branch. Please review. Thanks!

It's a cherry-pick of #242 and #244 and replaces #243.

[DrDaveD]

For @jthiltges to merge and deploy.

[jthiltges]

Changes published as revision 132

[DrDaveD]

There's more to do to get this onto other stratum 1s and other config distributions. Is someone following up on that?

[DrDaveD]

There's more to do to get this onto other stratum 1s and other config distributions. Is someone following up on that?

What's the status of the followup beyond osg?

[stebo85]

Dear @DrDaveD - thank you for following up on this :) You can't imagine how helpful these Stratum 1 servers are for our project and I am very grateful that this community exists with all the people working hard in the background to make this possible!

Is there anything I can help?

I started adding the non-CDN servers to our check (https://github.com/NeuroDesk/neurodesktop/actions/runs/8843560012/job/24283952855)

and so far these servers are already working: cvmfs-s1goc.opensciencegrid.org sampacs01.if.usp.br cvmfs-s1fnal.opensciencegrid.org

These are not yet working: cvmfs01.nikhef.nl cvmfs-egi.gridpp.rl.ac.uk cvmfs-stratum-one.ihep.ac.cn cvmfs-s1.hpc.swin.edu.au vmfs-s1bnl.opensciencegrid.org

Is there a map with a rough location of these so I can configure our geolocation DNS addresses cvmfs1/2/3.neurodesk.org for our users?

Are there costs for someone when we use the CDN addresses? Should we use them? What are the advantages/disadvantages?

[biozit]

Should I monitor this cache?

[DrDaveD]

@biozit there's no OSDF cache involved with this repository, so nothing for you to monitor.

@stebo85 I don't know of a map I can point you to, but the country codes are a good hint, or you can look up their geo location from their IP addresses on various places on the internet. The opensciencegrid.org servers are in the U.S.

OSG operations people are working with the BNL administrator to add the repository to their stratum 1. That process is not automated like it is for FNAL and SAMPA. For the rest of the servers, someone needs to go through the process documented at the very end of the section https://osg-htc.org/docs/data/external-oasis-repos/#hosting-a-repository-on-oasis, opening a GGUS ticket following EGI's PROC20 linked there.

The CDN does not cost us any money because we use the Cloudflare free tier. We want small sites that do not have their own squid caches to use the CDN. That gives them better performance and reduces load on the stratum 1 servers. We ask all larger sites to have their own squid caches, and we prefer for them to not go through the CDN because we can monitor their behavior much better if they connect directly to the stratum 1s. If larger sites do not have their own squid caches and instead use the CDN, it not only is an unnecessary strain on their Wide Area Network with the many small transfers, it also increases the use of the CDN. Although we cleared our use case with Cloudflare representatives years ago, we would rather not overuse the CDN in case we someday raise red flags there. So far we have typically used around 150TB of Cloudflare bandwidth per month for all of openhtc.io, and around 10% of that traffic is forwarded to our servers.

[stebo85]

Dear Dave,

Thank you for the explanation. That makes all sense!

Should I be the person who opens the GGUS ticked and foolow the EGI's PROC20 document?

Thank you Steffen

[DrDaveD]

Possibly, or someone else from the Neurodesk project. I believe EGI will want there to be at least two of their registered sites to be users of the repository. I don't know if the European users of Neurodesk are EGI registered sites or not.

[stebo85]

ok, I will contact EGI and ask. Thank you!