search

DLHub-Argonne / dlhub_sdk

Python Interface to the Data and Learning Hub for Science
Apache License 2.0
27 stars 6 forks source link

update DLHub SDK for compatibility w/ funcx->globus_compute transition #186

Closed ericblau closed 1 year ago

ericblau commented 1 year ago

The most recent version of funcx on PyPi is no longer functional; attempts to use it w/ the most recent DLHub SDK produce:

UserWarning: The funcX SDK has been renamed to Globus Compute SDK and the new package is available on PyPI: https://pypi.org/project/globus-compute-sdk/

Please consider upgrading to Globus Compute. More information can be found at: https://globus-compute.readthedocs.io/en/latest/funcx_upgrade.html

warnings.warn(DEPRECATION_FUNCX) Traceback (most recent call last): File "/srv/conda/envs/notebook/bin/funcx-manager", line 5, in from funcx_endpoint.executors.high_throughput.funcx_manager import cli_run File "/srv/conda/envs/notebook/lib/python3.7/site-packages/funcx_endpoint/executors/init.py", line 1, in from funcx_endpoint.executors.high_throughput.executor import HighThroughputExecutor File "/srv/conda/envs/notebook/lib/python3.7/site-packages/funcx_endpoint/executors/high_throughput/executor.py", line 26, in from funcx_endpoint.executors.high_throughput import interchange, zmq_pipes File "/srv/conda/envs/notebook/lib/python3.7/site-packages/funcx_endpoint/executors/high_throughput/interchange.py", line 22, in from funcx.sdk.client import FuncXClient ModuleNotFoundError: No module named 'funcx.sdk.client'

Thus, we need to update all our imports to use globus_compute instead.

ericblau commented 1 year ago

My take on this is that we should make all necessary changes to be able to use the globus-compute-sdk package instead of the funcx sdk package, but that no other references to funcx should be altered. That is to say, the schema for dlhub.json should continue to use "funcx_id", the DLHubClient method clear_funcx_cache should remain as-is, etc. To do otherwise would require DLHub SDK users to change their code, which seems unnecessary. @blaiszik @ascourtas do you agree?

ascourtas commented 1 year ago

maybe have 6 months before funcx <2 is deprecated

ascourtas commented 1 year ago

time box to 2 hours -- if it takes longer, bump

ericblau commented 1 year ago

Was almost able to only change the imports; the only other changes were in dlhub_sdk/utils/funcx_login_manager.py where the method "get_funcx_web_client" had to change to "get_web_client", and in the pytest "test_submit" where a function being mocker patched had to be updated.