search

eqasim-org / ile-de-france

An open synthetic population of Île-de-France for agent-based transport simulation
GNU General Public License v2.0
47 stars 69 forks source link

fix: update pyarrow to latest version #241

Closed sebhoerl closed 3 months ago

sebhoerl commented 3 months ago

For some reason, the environment runs well on Linux and Windows in CI, but it cannot be resolved on our Linux servers. Trying to update pyarrow here to check if it solves the problem (the indicated configuration works on our servers now). I have the feeling that the update of sqlite changed some unit tests before. Let's see what the automated tests say.

sebhoerl commented 3 months ago

@vincent-leblond Could you please check if this works for you? It is a bit a mystery for my why the environment.yml you pushed worked for you and both in Linux and on Windows in the unit tests here, but not on our servers. Now I just updated it to the latest version of pyarrow that is available on conda-forge and it seems to work on our servers and on Github.

vincent-leblond commented 3 months ago

@sebhoerl After creating a conda environment with the environment.yml, I was not able to make a complete run. Here is the error in my linux terminal. This error is pretty strange.

Capture d’écran du 2024-06-12 15-09-16

However, I am not fully comfortable with conda. I am more used to requirements.txt way. This is what we use in our python projects at Tellae. I have then installing packages in a python environment. My previous failed run was now successfull. I can't figure out why. If you want to try on your server, and unit tests, here is the content of my requirements.txt file (use pip install -r requirements.txt and eventually pip --no-cache-dir install -r requirements.txt). Please notice that pytables is replaced by tables and sqlite is commented.

matplotlib==3.7.1
pandas==1.5.3
scipy==1.10.1
numpy==1.23.5
geopandas==0.12.2
numba==0.56.4
palettable==3.3.0
scikit-learn==1.2.2
shapely==2.0.2
tqdm==4.65.0
tables==3.9.2
xlrd==2.0.1
openpyxl==3.1.0
py7zr==0.20.8
pytest==7.2.2
xlwt==1.3.0
fiona==1.9.2
#sqlite==3.46.0
synpp==1.5.1
pyarrow==16.1.0
mock==5.1.0
sebhoerl commented 3 months ago

Hm, ok, so if it works in the unit tests, I think I will merge it. In your log, what I can see is that the stack trace starts in conda (/miniconda3), then it loads the module from /home/vincent/tellae/Codes, but then it makes use of some numpy that is not located in the miniconda environment, but in /home/vincent/.local/lib. Not sure why this is, but it is rather a system set-up.