search

hep-gc / shoal

A squid cache publishing and advertising tool designed to work in fast changing environments
Apache License 2.0
4 stars 8 forks source link

shoal server can't be installed #144

Open MarcusEbert opened 6 years ago

MarcusEbert commented 6 years ago

Going through the install instructions in the shoal-server readme, I am not able to install a new shoal server.

The install instructions need to get fixed and tested.

DrDaveD commented 5 years ago

I am able to install on centos6 if I do yum --enablerepo=shoal-dev.

On centos6 without that extra repo, it attempts to install shoal-server-0.6.1-1 and runs into missing dependencies of pika>=0.9.13, web.py>=0.3, and pygeoip>=0.2.5.

On centos7, there's no shoal-server available in either the standard shoal repo or shoal-dev.

Can these yum repositories please be brought up to date?

DrDaveD commented 5 years ago

Oh, I take it back, the shoal-dev install on centos6 also had a problem after I let the install complete:

warning: rpmts_HdrFromFdno: Header V4 DSA/SHA1 Signature, key ID b4833a14: NOKEY

Public key for web.py-0.37-1.noarch.rpm is not installed
DrDaveD commented 5 years ago

Likewise for pygeoip-0.3.0-1.noarch.rpm. If I install those two by hand with rpm, ignoring signature, I get conflicts for the pika package with python-pika-0.9.5-2.el6.noarch. That was only installed because of a requirement for shoal-agent, and I don't need that on the same machine as shoal-server, so I deleted it. Then I could yum install --enablerepo=shoal-dev shoal-server. However there was a warning during install:

No handlers could be found for logger "shoal_server"
warning: %post(shoal-server-0.7.5-1.el6.noarch) scriptlet failed, exit status 1
Non-fatal POSTIN scriptlet failure in rpm package shoal-server-0.7.5-1.el6.noarch
MarcusEbert commented 5 years ago

@DrDaveD We look into it. If you want to use shoal for grid based squids serving /cvmfs, may I suggest that you use, in the meantime or in addition to an own server, the central shoal server we set up? That way WLCG grid workers can have a central point to get access to a nearby squid. If you want to give it a try, the server is at http://shoal.heprc.uvic.ca/ and you will see your squids in the list when they registered successful.

DrDaveD commented 5 years ago

My goal is to integrate a shoal server into the WLCG WPAD service so that's not going to be good enough for me. I'd like http://wlcg-wpad.cern.ch/wpad.dat (and http://wlcg-wpad.fnal.gov/wpad.dat) to return either the pre-configured squids that are now returned or dynamically registered squids, whichever is applicable.

I'm also thinking of including shoal-agent in the frontier-squid package, or at least distributing them together, and making it trivial for someone to configure the squid to register itself in the WLCG shoal server.

MarcusEbert commented 5 years ago

I see. I think we should try to have one WLCG shoal server where all squids register, no matter which experiment they support (most likely more than 1 anyway). Since CERN, DESY and others already register with our shoal server it would be good to keep it. For your use case, would it work to have us add an option to give an IP used to determine the nearest squids, e.g. http://shoal.heprc.uvic.ca/wpad.dat?$IP to return the squids in the wpad format (which it already does), but based on $IP and not on the IP that the query came from? Then clients can connect to your server, which collects the information from the static wpad system and from the shoal server, and sends back an appropriate combined list.

In any case, we are looking into the install issues you mentioned and let you know when it is fixed. It would be good to distributed the shoal-agent and the frontier-squid together or have the shoal-agent included in the frontier-squid package.

DrDaveD commented 5 years ago

I don't think that's an architecture scalable to the levels I have in mind. The existing wlcg-wpad service has 4 physical machines, 2 at CERN and 2 at FNAL (the machines also have other purposes as backup squid proxies when site squids fail), and they all have a copy of the squid information locally so they can immediately respond to all requests. I more have in mind a dump of the entire internal database of registered squids, downloaded every 5 minutes or so. Could you add an option for that?

I agree it would be good to have a single registration point for the WLCG. I assumed I would have the central registration point be at CERN, probably called wlcg-shoal.cern.ch, which I would make to be a DNS alias for wlcg-squid-monitor.cern.ch. That is currently one of a pair of machines that back each other up, and the backup can automatically take over for the primary if the primary goes down. It is already the source of data for the 4 wlcg-wpad machines. Then I would ask all the existing shoal-agents on the WLCG to switch to registering there.

Having said that, it might be fine to change that plan to have everybody register at shoal.heprc.uvic.ca, as long as wlcg-squid-monitor.cern.ch can periodically get a dump of the list of registered squids and their associated metadata. I prefer a json format, and we could negotiate the specific format.

MarcusEbert commented 5 years ago

Well, the dump of everything that is available in shoal is already there: http://shoal.heprc.uvic.ca/all gives you a list of all registered squids and their information. Is that what you need (except for the format maybe)?

But I'm not sure I understand what you want to do with the information as a full dump? Do you want to ingest this information into the wpad system then, or how does a client that queries the system would then get a list that is ordered by distance? Or are you planning to do your own GeoIP matching on top of wpad and shoal using the combined information of both?

DrDaveD commented 5 years ago

That should do. It has more than I need, but it includes what I need. You don't mind if I start reading that every 5 minutes? If you added a Last-Modified header I could have it use that time for the file and request with If-Modified-Since to avoid redownloads if nothing has changed.

Yes I plan to have wlcg-wpad do its own GeoIP matching, as it already does. If it doesn't find a match in the static WLCG data (which basically comes from the CMS & ATLAS site frontier configurations) I intend to have it also look in the shoal data. It currently does not use longitude & latitude; it uses the GeoIP organization from the GeoIP2-ISP database, a purchased maxmind database, and matches squid & client IPs by organization.

MarcusEbert commented 5 years ago

Should be no problem to query it every 5 minutes.