Closed meteorologist15 closed 4 months ago
In esgpull/models/file.py, adding a "Waiting = 'waiting'" line:
class FileStatus(Enum):
New = "new"
Queued = "queued"
Waiting = "waiting"
Starting = "starting"
Started = "started"
Pausing = "pausing"
Paused = "paused"
Error = "error"
Cancelled = "cancelled"
Done = "done"
@classmethod
def retryable(cls) -> list[FileStatus]:
return [cls.Error, cls.Cancelled]
and adding a "Waiting" entry for Enum in esgpull/migrations/versions/0.3.0_update_tables.py:
sa.Column(
"status",
sa.Enum(
"New",
"Queued",
"Waiting",
"Starting",
"Started",
"Pausing",
"Paused",
"Error",
"Cancelled",
"Done",
name="filestatus",
),
nullable=False,
),
seemed to fix the issue for me. I may submit a pull request with these proposed changes. Thanks.
Thanks for the report, I must have missed this status value somehow.
Your fix is indeed valid for your issue with database import, although the files with waiting
status are then never going to be used by esgpull. The waiting
files can only be "unlocked" by running esgpull retry waiting
(it sets them to queued
), but I don't think the retry
command should serve this purpose.
Maybe for a more functional fix, we would need to add some interpretation of synda's status values in esgpull/models/synda_file.py
inside the to_file()
method. I had included the status values from synda for a smooth transition (and obviously missed waiting
) but really only a few values are currently used by esgpull:
And since there were very few to no explanations on the meaning of each status value in synda's documentation, I did not try to add any interpretation in the previous releases. If you know more, could you explain what it means for a file to be waiting
in the context of synda, and maybe how it got to be in that state?
In any case, I'll look into the other issue you describe, the import hanging on error, it shouldn't happen.
Hm, after a quick look back to legacy synda code, I realized waiting
is simply the same as queued
, I must have forgot we renamed it. I should be able to fix this issue easily then.
My old Synda database contained files in a "waiting" status, which I believed got flagged when trying to do an import. The import appeared to work (the CLI interface displayed 100%), but afterwards, hung. I looked at the log and got the following:
I could always simply try removing the 'waiting' status from files in my existing Synda database, but I presume this error status as a whole is unexpected.