search

PySlurm / pyslurm

Python Interface to Slurm
https://pyslurm.github.io
GNU General Public License v2.0
474 stars 116 forks source link

Any plans to support Slurm 23 ? #274

Closed subhc closed 1 year ago

subhc commented 1 year ago

Details

Issue

Not an issue :) Thank you for this application, it is really easy to use and works so well! Recently the cluster got upgraded and pyslurm does not support 23.02 as of yet. Curious if there are any plans to support this version?

I tried installing the main branch, it's not compatible.

Processing /scratch/shared/beegfs/subha/projects/pyslurm
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: pyslurm
  Building wheel for pyslurm (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for pyslurm (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [144 lines of output]
      INFO: Found Slurm shared library in /usr/lib64
      INFO: Found Slurm header in /usr/include/slurm
      INFO: Detected Slurm version - 23.02
      INFO: Building PySlurm from source...
      INFO: Checking for objects to clean
      INFO: Removing build/
      INFO: Removing: pyslurm/pyslurm.c
      INFO: cleanup done
      Compiling pyslurm/pyslurm.pyx because it changed.
      [1/1] Cythonizing pyslurm/pyslurm.pyx
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.10
      creating build/lib.linux-x86_64-3.10/pyslurm
      copying pyslurm/__init__.py -> build/lib.linux-x86_64-3.10/pyslurm
      copying pyslurm/__version__.py -> build/lib.linux-x86_64-3.10/pyslurm
      running build_ext
      building 'pyslurm.pyslurm' extension
      creating build/temp.linux-x86_64-3.10
      creating build/temp.linux-x86_64-3.10/pyslurm
      gcc -pthread -B /users/subha/software/miniconda3/envs/vggbot/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /users/subha/software/miniconda3/envs/vggbot/include -fPIC -O2 -isystem /users/subha/software/miniconda3/envs/vggbot/include -fPIC -Ipyslurm -I/usr/include -I. -I/users/subha/software/miniconda3/envs/vggbot/include/python3.10 -c pyslurm/pyslurm.c -o build/temp.linux-x86_64-3.10/pyslurm/pyslurm.o
      pyslurm/pyslurm.c: In function ‘__pyx_f_7pyslurm_7pyslurm_6config___get’:
      pyslurm/pyslurm.c:15421:86: error: ‘slurm_conf_t’ has no member named ‘launch_type’
           __pyx_t_1 = __pyx_f_7pyslurm_5slurm_stringOrNone(__pyx_v_self->__pyx___Config_ptr->launch_type, __pyx_kp_u_); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 644, __pyx_L1_error)
                                                                                            ^
      pyslurm/pyslurm.c:15433:86: error: ‘slurm_conf_t’ has no member named ‘launch_type’
           __pyx_t_1 = __pyx_f_7pyslurm_5slurm_stringOrNone(__pyx_v_self->__pyx___Config_ptr->launch_type, __pyx_kp_u_); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 645, __pyx_L1_error)
                                                                                            ^
      pyslurm/pyslurm.c:16557:86: error: ‘slurm_conf_t’ has no member named ‘slurmctld_plugstack’
           __pyx_t_1 = __pyx_f_7pyslurm_5slurm_stringOrNone(__pyx_v_self->__pyx___Config_ptr->slurmctld_plugstack, __pyx_kp_u_); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 741, __pyx_L1_error)
                                                                                            ^
      pyslurm/pyslurm.c: In function ‘__pyx_pymod_exec_pyslurm’:
      pyslurm/pyslurm.c:85637:44: error: ‘ESLURM_TASKDIST_ARBITRARY_UNSUPPORTED’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(ESLURM_TASKDIST_ARBITRARY_UNSUPPORTED); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 59, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:85637:44: note: each undeclared identifier is reported only once for each function it appears in
      pyslurm/pyslurm.c:85649:44: error: ‘ESLURM_TASKDIST_REQUIRES_OVERCOMMIT’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(ESLURM_TASKDIST_REQUIRES_OVERCOMMIT); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 60, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:85913:44: error: ‘ESLURM_INVALID_BLOCK_STATE’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(ESLURM_INVALID_BLOCK_STATE); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 82, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:85925:44: error: ‘ESLURM_INVALID_BLOCK_LAYOUT’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(ESLURM_INVALID_BLOCK_LAYOUT); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 83, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:85937:44: error: ‘ESLURM_INVALID_BLOCK_NAME’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(ESLURM_INVALID_BLOCK_NAME); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 84, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:86657:44: error: ‘ESLURMD_PIPE_ERROR_ON_TASK_SPAWN’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(ESLURMD_PIPE_ERROR_ON_TASK_SPAWN); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 144, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:86717:44: error: ‘ESLURMD_UID_NOT_FOUND’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(ESLURMD_UID_NOT_FOUND); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 149, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:86729:44: error: ‘ESLURMD_GID_NOT_FOUND’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(ESLURMD_GID_NOT_FOUND); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 150, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:86789:44: error: ‘ESLURMD_MODIFY_BATCH_DIR_ERROR’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(ESLURMD_MODIFY_BATCH_DIR_ERROR); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 155, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:86801:44: error: ‘ESLURMD_CREATE_BATCH_SCRIPT_ERROR’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(ESLURMD_CREATE_BATCH_SCRIPT_ERROR); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 156, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:86813:44: error: ‘ESLURMD_MODIFY_BATCH_SCRIPT_ERROR’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(ESLURMD_MODIFY_BATCH_SCRIPT_ERROR); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 157, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:86837:44: error: ‘ESLURMD_SHARED_MEMORY_ERROR’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(ESLURMD_SHARED_MEMORY_ERROR); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 159, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:86861:44: error: ‘ESLURMD_SET_SID_ERROR’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(ESLURMD_SET_SID_ERROR); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 161, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:86873:44: error: ‘ESLURMD_CANNOT_SPAWN_IO_THREAD’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(ESLURMD_CANNOT_SPAWN_IO_THREAD); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 162, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:86885:44: error: ‘ESLURMD_FORK_FAILED’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(ESLURMD_FORK_FAILED); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 163, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:86945:44: error: ‘ESLURMD_SESSION_KILLED’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(ESLURMD_SESSION_KILLED); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 168, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:87029:44: error: ‘ESCRIPT_CHDIR_FAILED’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(ESCRIPT_CHDIR_FAILED); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 175, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:87041:44: error: ‘ESCRIPT_OPEN_OUTPUT_FAILED’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(ESCRIPT_OPEN_OUTPUT_FAILED); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 176, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:87053:44: error: ‘ESCRIPT_NON_ZERO_RETURN’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(ESCRIPT_NON_ZERO_RETURN); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 177, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:87065:44: error: ‘SLURM_PROTOCOL_SOCKET_IMPL_ZERO_RECV_LENGTH’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(SLURM_PROTOCOL_SOCKET_IMPL_ZERO_RECV_LENGTH); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 178, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:87077:44: error: ‘SLURM_PROTOCOL_SOCKET_IMPL_NEGATIVE_RECV_LENGTH’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(SLURM_PROTOCOL_SOCKET_IMPL_NEGATIVE_RECV_LENGTH); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 179, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:87089:44: error: ‘SLURM_PROTOCOL_SOCKET_IMPL_NOT_ALL_DATA_SENT’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(SLURM_PROTOCOL_SOCKET_IMPL_NOT_ALL_DATA_SENT); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 180, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:87149:44: error: ‘ESLURM_AUTH_FOPEN_ERROR’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(ESLURM_AUTH_FOPEN_ERROR); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 185, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:87161:44: error: ‘ESLURM_AUTH_NET_ERROR’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(ESLURM_AUTH_NET_ERROR); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 186, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:87173:44: error: ‘ESLURM_AUTH_UNABLE_TO_SIGN’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(ESLURM_AUTH_UNABLE_TO_SIGN); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 187, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:87197:44: error: ‘ESLURM_AUTH_MEMORY’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(ESLURM_AUTH_MEMORY); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 189, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:87209:44: error: ‘ESLURM_AUTH_INVALID’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_slurm_err_t(ESLURM_AUTH_INVALID); if (unlikely(!__pyx_t_2)) __PYX_ERR(6, 190, __pyx_L1_error)
                                                  ^
      pyslurm/pyslurm.c:89066:40: error: ‘PRIVATE_CLOUD_NODES’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_uint8_t(PRIVATE_CLOUD_NODES); if (unlikely(!__pyx_t_2)) __PYX_ERR(7, 160, __pyx_L1_error)
                                              ^
      pyslurm/pyslurm.c:90890:41: error: ‘DEBUG_FLAG_ESEARCH’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_uint64_t(DEBUG_FLAG_ESEARCH); if (unlikely(!__pyx_t_2)) __PYX_ERR(7, 332, __pyx_L1_error)
                                               ^
      pyslurm/pyslurm.c:92762:55: error: ‘WAIT_BLOCK_MAX_ERR’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_enum__job_state_reason(WAIT_BLOCK_MAX_ERR); if (unlikely(!__pyx_t_2)) __PYX_ERR(8, 55, __pyx_L1_error)
                                                             ^
      pyslurm/pyslurm.c:92774:55: error: ‘WAIT_BLOCK_D_ACTION’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_enum__job_state_reason(WAIT_BLOCK_D_ACTION); if (unlikely(!__pyx_t_2)) __PYX_ERR(8, 56, __pyx_L1_error)
                                                             ^
      pyslurm/pyslurm.c:94958:58: error: ‘SELECT_JOBDATA_PAGG_ID’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_enum__select_jobdata_type(SELECT_JOBDATA_PAGG_ID); if (unlikely(!__pyx_t_2)) __PYX_ERR(8, 263, __pyx_L1_error)
                                                                ^
      pyslurm/pyslurm.c:94970:58: error: ‘SELECT_JOBDATA_PTR’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_enum__select_jobdata_type(SELECT_JOBDATA_PTR); if (unlikely(!__pyx_t_2)) __PYX_ERR(8, 264, __pyx_L1_error)
                                                                ^
      pyslurm/pyslurm.c:94982:58: error: ‘SELECT_JOBDATA_CLEANING’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_enum__select_jobdata_type(SELECT_JOBDATA_CLEANING); if (unlikely(!__pyx_t_2)) __PYX_ERR(8, 265, __pyx_L1_error)
                                                                ^
      pyslurm/pyslurm.c:95006:58: error: ‘SELECT_JOBDATA_RELEASED’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_enum__select_jobdata_type(SELECT_JOBDATA_RELEASED); if (unlikely(!__pyx_t_2)) __PYX_ERR(8, 267, __pyx_L1_error)
                                                                ^
      pyslurm/pyslurm.c:97046:40: error: ‘CLUSTER_FLAG_A1’ undeclared (first use in this function)
         __pyx_t_2 = __Pyx_PyInt_From_uint8_t(CLUSTER_FLAG_A1); if (unlikely(!__pyx_t_2)) __PYX_ERR(9, 63, __pyx_L1_error)
                                              ^
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pyslurm
Failed to build pyslurm
ERROR: Could not build wheels for pyslurm, which is required to install pyproject.toml-based projects
tazend commented 1 year ago

Hi @subhc

yes, the current main branch is for 22.05, so compilation will fail. I'll try to check out the necessary changes for migrating to 23.02 in the coming days :)

subhc commented 1 year ago

Thanks for the quick response. Looking forward to the release :)

tazend commented 1 year ago

Hi, support for 23.02 is out now :) https://github.com/PySlurm/pyslurm/releases/tag/v23.2.0

subhc commented 1 year ago

Thank you!