dmwm / CRABClient

runrange
14 stars 35 forks source link

Make sure checkwrite create unique LFN path #5296

Closed novicecpp closed 4 months ago

novicecpp commented 4 months ago

In CI, we run CV/CCV in parallel to reduce the feedback time.

I usually have this error message when CI submit jobs (Runner run at 4 concurrents):

gfal-copy error: 17 (File exists) - Destination davs://eoscms.cern.ch:443/eos/cms/store/user/tseethon/crab3checkwrite_20240305_132750/crab3checkwrite_20240305_132750.tmp exists and overwrite is not set

This happened because current code rely on uniqueness of timestamp at the second level, which is not enough. Here is how crab checkwrite construct LFN:

        timestamp = str(time.strftime("%Y%m%d_%H%M%S"))
        self.filename = 'crab3checkwrite_' + timestamp  + '.tmp'
        self.subdir = 'crab3checkwrite_' + timestamp
        lfn = self.lfnPrefix + '/' + self.subdir + '/' + self.filename

As per your suggestion in MM, appending the uuid.uuid4() to self.filename should be enough.

belforte commented 4 months ago

Indeed there is no task name available to guarantee uniqueness when we do crab checkwrite