Earth-Information-System / fireatlas

https://earth-information-system.github.io/fireatlas/docs/
7 stars 3 forks source link

Python 3.13 breaks MAAP DPS build #164

Open zebbecker opened 1 month ago

zebbecker commented 1 month ago

Python 3.13 was released last week. Our run_dps_build.sh script was installing 3.13 during the DPS image build process, and some interaction between that and our requirements (scalene?) causes the DPS image build to fail. This is reproducible on ADE as of today.

to reproduce:

# clone fireatlas, run from repo base dir 
conda create --name fire_env 
conda activate fire_env
conda install pip # this command was installing python 3.13 and its corresponding pip version 24.2 
pip install -e .

error trace from DPS image build:

#25 [21/21] RUN bash fireatlas/maap_runtime/run_dps_build.sh && conda clean -afy
#25 0.429 Basedir: /app/fireatlas/maap_runtime
#25 0.430 Initial working directory: /app
#25 0.431 conda: /opt/conda/bin/conda
#25 0.433 Python: /opt/conda/bin/python
#25 0.435 Python 3.11.9
#25 0.435 /app/fireatlas/maap_runtime /app
#25 1.039 Channels:
#25 1.039  - defaults
#25 1.039 Platform: linux-64
#25 1.039 Collecting package metadata (repodata.json): ...working... done
#25 4.062 Solving environment: ...working... done
#25 4.073 
#25 4.073 ## Package Plan ##
#25 4.073 
#25 4.073   environment location: /opt/conda/envs/fire_env
#25 4.073 
#25 4.073 
#25 4.073 
#25 4.073 Proceed ([y]/n)? 
#25 4.073 Preparing transaction: ...working... done
#25 4.073 Verifying transaction: ...working... done
#25 4.093 Executing transaction: ...working... done
#25 4.095 #
#25 4.095 # To activate this environment, use
#25 4.095 #
#25 4.095 #     $ conda activate fire_env
#25 4.095 #
#25 4.095 # To deactivate an active environment, use
#25 4.095 #
#25 4.095 #     $ conda deactivate
#25 4.095 
#25 4.274 Activated fire_env
#25 4.894 Channels:
#25 4.894  - defaults
#25 4.894 Platform: linux-64
#25 4.894 Collecting package metadata (repodata.json): ...working... done
#25 5.114 Solving environment: ...working... done
#25 5.216 
#25 5.216 ## Package Plan ##
#25 5.216 
#25 5.216   environment location: /opt/conda/envs/fire_env
#25 5.216 
#25 5.216   added / updated specs:
#25 5.216     - pip
#25 5.216 
#25 5.216 
#25 5.216 The following packages will be downloaded:
#25 5.216 
#25 5.216     package                    |            build
#25 5.216     ---------------------------|-----------------
#25 5.216     _libgcc_mutex-0.1          |             main           3 KB
#25 5.216     _openmp_mutex-5.1          |            1_gnu          21 KB
#25 5.216     bzip2-1.0.8                |       h5eee18b_6         262 KB
#25 5.216     ca-certificates-2024.9.24  |       h06a4308_0         130 KB
#25 5.216     expat-2.6.3                |       h6a678d5_0         [176](https://repo.maap-project.org/root/register-job-hysds-v4/-/jobs/15044#L176) KB
#25 5.216     ld_impl_linux-64-2.40      |       h12ee557_0         710 KB
#25 5.216     libffi-3.4.4               |       h6a678d5_1         141 KB
#25 5.216     libgcc-ng-11.2.0           |       h1234567_1         5.3 MB
#25 5.216     libgomp-11.2.0             |       h1234567_1         474 KB
#25 5.216     libmpdec-4.0.0             |       h5eee18b_0          86 KB
#25 5.216     libstdcxx-ng-11.2.0        |       h1234567_1         4.7 MB
#25 5.216     libuuid-1.41.5             |       h5eee18b_0          27 KB
#25 5.216     ncurses-6.4                |       h6a678d5_0         914 KB
#25 5.216     openssl-3.0.15             |       h5eee18b_0         5.2 MB
#25 5.216     pip-24.2                   |  py313h06a4308_0         2.4 MB
#25 5.216     python-3.13.0              |hf623796_100_cp313        36.0 MB
#25 5.216     python_abi-3.13            |          0_cp313           6 KB
#25 5.216     readline-8.2               |       h5eee18b_0         357 KB
#25 5.216     setuptools-75.1.0          |  py313h06a4308_0         1.9 MB
#25 5.216     sqlite-3.45.3              |       h5eee18b_0         1.2 MB
#25 5.216     tk-8.6.14                  |       h39e8969_0         3.4 MB
#25 5.216     tzdata-[202](https://repo.maap-project.org/root/register-job-hysds-v4/-/jobs/15044#L202)4b               |       h04d1e81_0         115 KB
#25 5.216     wheel-0.44.0               |  py313h06a4308_0         138 KB
#25 5.216     xz-5.4.6                   |       h5eee18b_1         643 KB
#25 5.216     zlib-1.2.13                |       h5eee18b_1         111 KB
#25 5.216     ------------------------------------------------------------
#25 5.216                                            Total:        64.5 MB
#25 5.[216](https://repo.maap-project.org/root/register-job-hysds-v4/-/jobs/15044#L216) 
#25 5.216 The following NEW packages will be INSTALLED:
#25 5.216 
#25 5.216   _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main 
#25 5.216   _openmp_mutex      pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu 
#25 5.216   bzip2              pkgs/main/linux-64::bzip2-1.0.8-h5eee18b_6 
#25 5.216   ca-certificates    pkgs/main/linux-64::ca-certificates-2024.9.24-h06a4308_0 
#25 5.216   expat              pkgs/main/linux-64::expat-2.6.3-h6a678d5_0 
#25 5.216   ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.40-h12ee557_0 
#25 5.216   libffi             pkgs/main/linux-64::libffi-3.4.4-h6a678d5_1 
#25 5.216   libgcc-ng          pkgs/main/linux-64::libgcc-ng-11.2.0-h1234567_1 
#25 5.216   libgomp            pkgs/main/linux-64::libgomp-11.2.0-h1[234](https://repo.maap-project.org/root/register-job-hysds-v4/-/jobs/15044#L234)567_1 
#25 5.216   libmpdec           pkgs/main/linux-64::libmpdec-4.0.0-h5eee18b_0 
#25 5.216   libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-11.2.0-h1234567_1 
#25 5.216   libuuid            pkgs/main/linux-64::libuuid-1.41.5-h5eee18b_0 
#25 5.216   ncurses            pkgs/main/linux-64::ncurses-6.4-h6a678d5_0 
#25 5.216   openssl            pkgs/main/linux-64::openssl-3.0.15-h5eee18b_0 
#25 5.216   pip                pkgs/main/linux-64::pip-24.2-py313h06a4308_0 
#25 5.216   python             pkgs/main/linux-64::python-3.13.0-hf6[237](https://repo.maap-project.org/root/register-job-hysds-v4/-/jobs/15044#L237)96_100_cp313 
#25 5.216   python_abi         pkgs/main/linux-64::python_abi-3.13-0_cp313 
#25 5.216   readline           pkgs/main/linux-64::readline-8.2-h5eee18b_0 
#25 5.216   setuptools         pkgs/main/linux-64::setuptools-75.1.0-py313h06a4308_0 
#25 5.216   sqlite             pkgs/main/linux-64::sqlite-3.45.3-h5eee18b_0 
#25 5.216   tk                 pkgs/main/linux-64::tk-8.6.14-h39e8969_0 
#25 5.216   tzdata             pkgs/main/noarch::tzdata-2024b-h04d1e81_0 
#25 5.216   wheel              pkgs/main/linux-64::wheel-0.44.0-py313h06a4308_0 
#25 5.216   xz                 pkgs/main/linux-64::xz-5.4.6-h5eee18b_1 
#25 5.216   zlib               pkgs/main/linux-64::zlib-1.2.13-h5eee18b_1 
#25 5.216 
#25 5.216 
#25 5.216 Proceed ([y]/n)? 
#25 10.50 
#25 10.50 Downloading and Extracting Packages: ...working... done
#25 10.50 Preparing transaction: ...working... done
#25 10.70 Verifying transaction: ...working... done
#25 12.38 Executing transaction: ...working... done
#25 13.21 Installed pip
#25 13.21 /opt/conda/envs/fire_env/bin/pip
#25 ERROR: process "/bin/sh -c bash ${BUILD_CMD} && conda clean -afy" did not complete successfully: exit code: 1
------
 > [21/21] RUN bash fireatlas/maap_runtime/run_dps_build.sh && conda clean -afy:
5.216   wheel              pkgs/main/linux-64::wheel-0.44.0-py313h06a4308_0 
5.216   xz                 pkgs/main/linux-64::xz-5.4.6-h5eee18b_1 
5.216   zlib               pkgs/main/linux-64::zlib-1.2.13-h5eee18b_1 
5.216 
5.216 
5.216 Proceed ([y]/n)? 
done
done
13.21 Installed pip
13.21 /opt/conda/envs/fire_env/bin/pip
------
Dockerfile:71
--------------------
  69 |         git checkout ${BRANCH} && \
  70 |         chmod -R 775 .
  71 | >>> RUN bash ${BUILD_CMD} && conda clean -afy
  72 |     
  73 |     CMD ["/bin/bash"]
--------------------
ERROR: failed to solve: process "/bin/sh -c bash ${BUILD_CMD} && conda clean -afy" did not complete successfully: exit code: 1
[ERROR] Failed to build docker container for: container-fireatlas:maap-build-troubleshoot
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit status 1
zebbecker commented 1 month ago

Looks like scalene pins just conflict with 3.13

fresh) root@workspacem0onea65hvi4h21y:~# conda install scalene
Channels:
 - conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: \ warning  libmamba Added empty dependency for problem type SOLVER_RULE_UPDATE
failed

LibMambaUnsatisfiableError: Encountered problems while solving:
  - nothing provides _python_rc needed by python-3.12.0rc3-rc3_hab00c5b_1_cpython

Could not solve for environment specs
The following packages are incompatible
├─ pin-1 is installable and it requires
│  └─ python 3.13.* , which can be installed;
└─ scalene is not installable because there are no viable options
   ├─ scalene [1.5.10|1.5.11|...|1.5.41] would require
   │  └─ python >=3.10,<3.11.0a0 , which conflicts with any installable versions previously reported;
   ├─ scalene [1.5.10|1.5.11|...|1.5.34] would require
   │  └─ python >=3.8,<3.9.0a0 , which conflicts with any installable versions previously reported;
   ├─ scalene [1.5.10|1.5.11|...|1.5.41] would require
   │  └─ python >=3.9,<3.10.0a0 , which conflicts with any installable versions previously reported;
   ├─ scalene [1.5.13|1.5.14|...|1.5.41] would require
   │  └─ python >=3.11,<3.12.0a0 , which conflicts with any installable versions previously reported;
   ├─ scalene 1.5.31.1 would require
   │  └─ python >=3.12.0rc3,<3.13.0a0  but there are no viable options
   │     ├─ python [3.12.0|3.12.1|...|3.12.7] conflicts with any installable versions previously reported;
   │     └─ python 3.12.0rc3 would require
   │        └─ _python_rc, which does not exist (perhaps a missing channel);
   └─ scalene [1.5.31.1|1.5.33.1|...|1.5.41] would require
      └─ python >=3.12,<3.13.0a0 , which conflicts with any installable versions previously reported.

Pins seem to be involved in the conflict. Currently pinned specs:
 - python 3.13.* (labeled as 'pin-1')