Closed rmanzoni closed 2 years ago
Thanks for reporting @rmanzoni
This is indeed an unintended conseuqnece. IMO, we should never delete files unless the user actively asks for it.
This deletion happens because create_files has an argument remove_old
, which currently defaults to true https://github.com/HEPData/hepdata_lib/blob/master/hepdata_lib/__init__.py#L541
This should be changed to defaulting to false, so that a user would have to explciitly call submission.create_files(....,remove_old=True)
in order for any deletion to take place.
Hi Andreas, thanks for the quick support! Just to avoid misunderstandings, the problem I observed is that all files that lived in the same directory as that of the hepdata script got deleted by it. Even those that had nothing to do with it. If it just deleted the existing hepdata outdir or the submission file that would've been minor. So I guess perhaps just changing the default is not enough, as it would reproduce the same bad behaviour when the user sets it to true.
We currently do not have a mechanism to track what files are created by hepdata-lib. Therefore, the straightforward implementation of remove_old
just removes the entire output directory. The entire reasoning here is that unrelated files in the output directory, which then go into the submission tar ball, will lead the hepdata validator to fail. For each file in the tar ball, it checks whether the file is referred to in the submission.yaml file. If not, it fails, arguing that something likely went wrong.
Therefore, full deletion of the output directory is sensible in principle. However, it should of course not delete unrelated files that are outside of its scope. It seems to me that changing the default output direcotry to something like ./hepdata_output/
would solve this. After all, a sensible user would not store unrelated in such a specifically named output directory. Of course the documentation of remove_old
should make obvious to the user that the entire folder is deleted, not just individual files.
I still have the impression that I didn't explain myself clear: I attach a screenshot of the directory that contains the plot I'm trying to include in the submission object using the hepdata.py
script that I've reported above.
When it gets to submission.create_files()
it deletes the entire content of the directory on display, not just output
. And then fails with the message I reported.
I would've been ok if it removed just output
and its content.
I agree it does not make sense to add spurious files in there.
Does the screenshot match the code you shared above? In the code you posted above, you call create_files()
without any arguments. This means that the default output directory is used, namely outdir="."
, i.e. the directory you are currently in. The code then tries to trigger deletion of outdir
. Based on the code you shared, I do not know what the output
directory is in the screenshot.
ok, now I understand.
Indeed it's called w/o arguments, so iiuc it defaults to '.'
, hence the problem I experienced.
I confirm that when I pass a different directory name all works as it should.
Thanks!
Thanks for reporting and providing further details. hepdata_lib
v0.10.1 that contains the hotfix is on PyPI. I've created #194 for discussion of a better/robust solution.
Hi!
this script based off the simple examples I found in the documentation, fails to write the submission file
submission.create_files()
and deletes all other files that reside in the same directory it is launched from.I guess this is not the intended behaviour...
this is the error it rreturns