search

SIGLA-GU / siglatools

Other
1 stars 1 forks source link

Prefect 2.X - Revert or Refactor? #67

Open nniiicc opened 4 months ago

nniiicc commented 4 months ago

@tohuynh - I tried to bump a few versions of prefect to resolve the action error in the manual link checker...

The dilemma is this:

I think the best solution is to refactor the prefect code to use flow or task instead ... But I don't want to mess around doing that unless you know of an easier way to continue using a 1.X version

tohuynh commented 4 months ago

Based on the logs, the difference between the one that succeeded and failed has something to do with pip install --upgrade pip setuptools wheel command in the workflow files. The failed run has a higher pip version. I'm not entirely sure, but I think the newer version of pip wasn't able to install prefect 1.2.4 because it thinks prefect has incorrect metadata. 

Requested prefect==1.2.4 from https://files.pythonhosted.org/packages/2b/07/019c52af7a0ba2de3a858b6868f1b44efca4e025bbf7ea91945a23634609/prefect-1.2.4-py3-none-any.whl (from siglatools==0.1.0) has invalid metadata: Expected matching RIGHT_PARENTHESIS for LEFT_PARENTHESIS, after version specifier
    kubernetes (>=9.0.0a1.0) ; extra == 'all_extras'
               ~~~~~~~~~~^
Please use pip<24.1 if you need to use this version.

What about we could do is pin the specific versions with 

pip install --upgrade pip==24.0 setuptools==70.0.0 wheel==0.43.0

From a previous successful run logs:

Successfully installed pip-24.0 setuptools-70.0.0 wheel-0.43.0

The thing that worries me is that in the future, a higher version of ubuntu-latest image might not be able to use these specific versions. But I think it's better than to refactor and retest these scripts and I haven't been following prefect's development. We could also pin the specific ubuntu image, but eventually that image will no longer have long-term support and we'd have to use a different one.

nniiicc commented 3 months ago

Totally agree - I think pin the image

tohuynh commented 3 months ago

OK I'll try to fix the issue on a separate branch first.