search

JuliaPy / pyjuliapkg

Manage your Julia dependencies from Python
MIT License
45 stars 12 forks source link

Possible concurrency issues with resolve on a cluster #19

Open amilsted opened 8 months ago

amilsted commented 8 months ago

I think I am seeing multiple processes trying to simultaneously resolve julia versions and dependencies, mirroring https://github.com/JuliaPy/CondaPkg.jl/issues/40. Maybe locking is needed here too?

cjdoris commented 8 months ago

Yeah makes sense.

amilsted commented 8 months ago

Oh, by the way, I wonder if "pidfile" locking makes sense on a cluster? I think it probably doesn't, since there is no global list of processes.

cjdoris commented 8 months ago

I just took a look at the source:

https://github.com/vtjnash/Pidfile.jl/blob/67c3f23b25d4e43f1f5785ce77c2cac6952fefcf/src/Pidfile.jl#L128

Pidfile.jl uses both the PID and hostname, which presumably means it works with shared filesystems too.

amilsted commented 8 months ago

Ah, that's smart! Much better than https://github.com/mosquito/python-pidfile/blob/master/pidfile/pidfile.py

https://github.com/keiranmraine/pidlock sees to use hostname too.

cjdoris commented 8 months ago

Oh yeah of course this is a python package so need a python implementation of pidfiles.

cjdoris commented 8 months ago

pidlock looks good thanks

amilsted commented 8 months ago

Btw, is it really necessary to write out the meta file every time? Perhaps we could check for changes first?