search

ganga-devs / ganga

Ganga is an easy-to-use frontend for job definition and management
GNU General Public License v3.0
100 stars 159 forks source link

Need gfal2 plugins #2305

Closed mesmith75 closed 8 months ago

mesmith75 commented 8 months ago

We have had a bit of a problem with the monitoring. Essentially when the monitoring needs to download oversized sandboxes from grid storage (not the Dirac server) it cannot find the gfal2 plugin. I guess as we are not using Dirac's python we are missing the necessary path. I have installed the plugin for our installation (I had to change from miniconda to micromamba to do it) but I wonder if there is a better way?

chrisburr commented 8 months ago

I guess as we are not using Dirac's python we are missing the necessary path.

Can you explain what you mean by this? Has the way ganga interacts with DIRAC changed?

egede commented 8 months ago

I guess as we are not using Dirac's python we are missing the necessary path.

Can you explain what you mean by this? Has the way ganga interacts with DIRAC changed?

Yes, it has changed. The calls to Dirac used to be done by a separate python process which gave rise to the very significant bottlenecks that we had in the interaction with Dirac for monitoring and download of information. We now do this in the main python process in Ganga via asynchronous calls. Performance is much better. However, this means that we do not use the python install area that comes with a DIRAC API installation. When DIRAC changes to a RESTful interface this point should hopefully no longer be there.

chrisburr commented 8 months ago

I guess this corrosponds to here? https://github.com/ganga-devs/ganga/blob/develop/ganga/GangaDirac/Lib/Server/DiracExecutorProcess.py

If so, that's going to be very fragile but I'll try to keep it in mind when we make changes to warn you in advance.

egede commented 8 months ago

@mesmith75 @joj0s Did we actually use this method in the end? I am a confused about what we eventually did.