Import Synda fails/hangs when files from Synda database are in a "waiting" state

[meteorologist15]

My old Synda database contained files in a "waiting" status, which I believed got flagged when trying to do an import. The import appeared to work (the CLI interface displayed 100%), but afterwards, hung. I looked at the log and got the following:

[2024-01-31 19:28:40]  DEBUG     root
Locals:
^M^[[2K{'cls': <enum 'FileStatus'>, 'value': 'waiting', 'exc': None, 'result': None, 've_exc': None}
Working... ^[[38;2;249;38;114m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━^[[0m^[[38;2;249;38;114m╸^[[0m ^[[35m100%^[[0m ^[[36m0:00:20^[[0m

[2024-01-31 19:28:40]  ERROR     root

Traceback (most recent call last):
  File "/net2/ker/anaconda3/envs/esgdownload/lib/python3.12/site-packages/esgpull/tui.py", line 164, in logging
    yield
  File "/net2/ker/anaconda3/envs/esgdownload/lib/python3.12/site-packages/esgpull/cli/self.py", line 237, in import_synda
    nb_imported = esg.import_synda(url=path, track=True, ask=True)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/net2/ker/anaconda3/envs/esgdownload/lib/python3.12/site-packages/esgpull/esgpull.py", line 230, in import_synda
    file = synda_file.to_file()
           ^^^^^^^^^^^^^^^^^^^^
  File "/net2/ker/anaconda3/envs/esgdownload/lib/python3.12/site-packages/esgpull/models/synda_file.py", line 66, in to_file
    status=FileStatus(self.status),
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/net2/ker/anaconda3/envs/esgdownload/lib/python3.12/enum.py", line 744, in __call__
    return cls.__new__(cls, value)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/net2/ker/anaconda3/envs/esgdownload/lib/python3.12/enum.py", line 1158, in __new__
    raise ve_exc
ValueError: 'waiting' is not a valid FileStatus

I could always simply try removing the 'waiting' status from files in my existing Synda database, but I presume this error status as a whole is unexpected.

[meteorologist15]

In esgpull/models/file.py, adding a "Waiting = 'waiting'" line:

class FileStatus(Enum):
    New = "new"
    Queued = "queued"
    Waiting = "waiting"
    Starting = "starting"
    Started = "started"
    Pausing = "pausing"
    Paused = "paused"
    Error = "error"
    Cancelled = "cancelled"
    Done = "done"

    @classmethod
    def retryable(cls) -> list[FileStatus]:
        return [cls.Error, cls.Cancelled]

and adding a "Waiting" entry for Enum in esgpull/migrations/versions/0.3.0_update_tables.py:

        sa.Column(
            "status",
            sa.Enum(
                "New",
                "Queued",
                "Waiting",
                "Starting",
                "Started",
                "Pausing",
                "Paused",
                "Error",
                "Cancelled",
                "Done",
                name="filestatus",
            ),  
            nullable=False,
        ),  

seemed to fix the issue for me. I may submit a pull request with these proposed changes. Thanks.

[svenrdz]

Thanks for the report, I must have missed this status value somehow.

Your fix is indeed valid for your issue with database import, although the files with waiting status are then never going to be used by esgpull. The waiting files can only be "unlocked" by running esgpull retry waiting (it sets them to queued), but I don't think the retry command should serve this purpose.

Maybe for a more functional fix, we would need to add some interpretation of synda's status values in esgpull/models/synda_file.py inside the to_file() method. I had included the status values from synda for a smooth transition (and obviously missed waiting) but really only a few values are currently used by esgpull:

• queued
• starting
• cancelled
• error
• done

And since there were very few to no explanations on the meaning of each status value in synda's documentation, I did not try to add any interpretation in the previous releases. If you know more, could you explain what it means for a file to be waiting in the context of synda, and maybe how it got to be in that state?

In any case, I'll look into the other issue you describe, the import hanging on error, it shouldn't happen.

[svenrdz]

Hm, after a quick look back to legacy synda code, I realized waiting is simply the same as queued, I must have forgot we renamed it. I should be able to fix this issue easily then.