Closed adonnini closed 1 year ago
hongyeehh
commented
1 year ago Hi adonnini1,
Thanks for pointing out the typo. This has already been corrected.
Unfortunately, I wasn't able to reproduce your error. I assume this is caused by the version numbers of your installed packages. Could you make sure to install the packages with the correct version number in the readme.md? Especially, use shapely<=1.8.5
adonnini
commented
1 year ago Hi Ye,
Thanks for getting back to me.
I compared the versions of the modules required by location-mode-prediction as listed in the readme.md page:
Python 3.9.12, Geopandas 0.10.2, trackintel 1.1.10, PyTorch 1.12.1, cudatoolkit 11.3 (I could not find anywhere in readme.md a mention of the required version of shapely)
with the versions installed on my system. They are all the same except for trackintel (your own software). The version installed on my system is the latest listed on your GitHub repository, 1.1.12. Does this make a difference?
I did as you suggested. I removed shapely 2.0.1 and installed shapely 1.8.5
Execution of python3 prepro/geolife.py geolife 20 failed. Please see error detail below.
I can't help but feel that I am missing/overlooking something obvious.
Unfortunately, it looks like I am really stuck as you are not able to reproduce the problem. Does anyone on your team use a Linux based system? That would be helpful. Otherwise, I guess I'll have to look for other ways to implement a neural network for my application.
Thanks for your help and responsiveness,
Alex
(base) adonnini1@actlnxlptp6:~/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction$ python3 prepro/geolife.py geolife 20
/home/adonnini1/anaconda3/lib/python3.9/site-packages/geopandas/_compat.py:123: UserWarning: The Shapely GEOS version (3.10.3-CAPI-1.16.1) is incompatible with the GEOS version PyGEOS was compiled with (3.10.4-CAPI-1.16.2). Conversions between both will be slow.
warnings.warn(
Traceback (most recent call last):
File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction/prepro/geolife.py", line 8, in
adonnini
commented
1 year ago Hi Ye,
Thanks for getting back to me.
I compared the versions of the modules required by location-mode-prediction as listed in the readme.md page:
Python 3.9.12, Geopandas 0.10.2, trackintel 1.1.10, PyTorch 1.12.1, cudatoolkit 11.3 (I could not find anywhere in readme.md a mention of the required version of shapely)
with the versions installed on my system. They are all the same except for trackintel (your own software). The version installed on my system is the latest listed on your GitHub repository, 1.1.12. Does this make a difference?
I did as you suggested. I removed shapely 2.0.1 and installed shapely 1.8.5
Execution of python3 prepro/geolife.py geolife 20 failed. Please see error detail below.
I can't help but feel that I am missing/overlooking something obvious.
Unfortunately, it looks like I am really stuck as you are not able to reproduce the problem. Does anyone on your team use a Linux based system? That would be helpful. Otherwise, I guess I'll have to look for other ways to implement a neural network for my application.
Thanks for your help and responsiveness,
Alex
(base)
@.:~/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction$
python3 prepro/geolife.py geolife 20
/home/adonnini1/anaconda3/lib/python3.9/site-packages/geopandas/_compat.py:123:
UserWarning: The Shapely GEOS version (3.10.3-CAPI-1.16.1) is
incompatible with the GEOS version PyGEOS was compiled with
(3.10.4-CAPI-1.16.2). Conversions between both will be slow.
warnings.warn(
Traceback (most recent call last):
File
"/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction/prepro/geolife.py",
line 8, in
Ye wrote:
Hi adonnini1,
Thanks for pointing out the typo. This has already been corrected.
Unfortunately, I wasn't able to reproduce your error. I assume this is caused by the version numbers of your installed packages. Could you make sure to install the packages with the correct version number in the readme.md? Especially, use |shapely<=1.8.5|
— Reply to this email directly, view it on GitHub https://github.com/mie-lab/location-mode-prediction/issues/1#issuecomment-1425427535, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG4GZIO5CNHNR4SXBLDO4SLWWX5XXANCNFSM6AAAAAAUWPYDUU. You are receiving this because you authored the thread.Message ID: @.***>
adonnini
commented
1 year ago Hi, I have done some more research on this problem. I found this https://stackoverflow.com/questions/70911227/dtype-timedelta64ns-cannot-be-converted-to-datetime64ns It does not necessarily apply. However, I did a search in the repository for datetime and timedelta. It turns out that both datetime and timedelta are used (only) in
/prepro)/prePro.py
Since the error is "ValueError: unknown type timedelta64[ns]"
I wonder if the issue has to do with the columns 6 and 7 in the geolife plt file where col. 6 is a date, and col 7 is a time and in geolife.py there is no code to manage dates and times as there is in prePro.py
I am probably completely off base...
hongyeehh
commented
1 year ago The preprocessing of Geolife data is implemented with trackintel, which is tested on all mac, Linux, and windows operating systems. Currently, I cannot reproduce your error, and following your bug report, I could not spot an error in trackintel. If you can be more specific on the error, please open an issue there.
I updated the environment file with support for newer package versions. Could you re-install the environment again and check if the error persists?
Alternatively, as prePro.py only deals with preprocessing of the dataset, I could send you the generated files and you can continue checking the main parts of the transformer prediction model.
adonnini
commented
1 year ago Thanks very much for sending me the pre-process output file. I appreciate it. I will continue to look for ways to resolve the problem. I will also update my environment and try to run geolife.py again. I have one favor to ask. Could I send you a sample dataset of the data generated by my app formatted like Geolife data and stored in the same directory structure, have you pre-process it, and send me the pre-process output? Thanks
hongyeehh
commented
1 year ago We do not help with data preprocessing. This relates to the data format and should be taken care of by yourself.
adonnini
commented
1 year ago I understand. I was not asking for help with data pre-processing. Since the pre-processing code is not working on my system, I was simply asking to run my dataset on a system where your code works and send me the output, simply a matter of running the geolife preprocess command and, if it completes successfully, send me the output. That is all. If it does not cpomplete successfully, just le me know. That would be the end of it. Thank you
On my system running the pipeline using the Geolife dataset fails. I would appreciate your help in resolving this issue. Thank you,
My environment: Lenovo ThinkPad T15g Gen 2i with 128 GB memory Debian Linux Bookworm Python 3.9
On my system,I have a directory called location-mode-prediction It contains: /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction/baselines /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction/config /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction/environment.yml /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction/fig /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction/geolife /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction/geolifeADonniniSample020223.zip /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction/geolifeContextQ /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction/geolifeOriginal /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction/geolifeOriginaTest /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction/LICENSE /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction/main.py /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction/main_baselines.py /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction/models /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction/paths.json /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction/prepro /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction/README.md /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction/requirements.txt /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction/utils /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction/.git /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction/.gitignore /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction/.vscode
The geolife directory contains the original Geolife dataset with the following directory structure:
./geolife/Data/xxx/*.plt
As per the readme.md insturctions, I created paths.json containing
{ "raw_geolife": "./geolife" }
As per previous messages to Ye, I had to change the following line
pfs, mode_labels = read_geolife(os.path.join(config[f"raw_geolife"], "data"), print_progress=True) to pfs, mode_labels = read_geolife(os.path.join(config[f"raw_geolife"], "Data"), print_progress=True)
as the path to the dataset is ./geolife/Data not ./geolife/data
When I run
python prePro/geolife.py geolife 20
in location-prediction-mode
it fails as follows (it fails in the same way when run with fewer user directories in the geolife/Data folder, and with my own test dataset):
(base) adonnini1@actlnxlptp6:~/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction$ python3 prepro/geolife.py geolife 20 /home/adonnini1/anaconda3/lib/python3.9/site-packages/geopandas/_compat.py:123: UserWarning: The Shapely GEOS version (3.11.1-CAPI-1.17.1) is incompatible with the GEOS version PyGEOS was compiled with (3.10.4-CAPI-1.16.2). Conversions between both will be slow. warnings.warn( /home/adonnini1/anaconda3/lib/python3.9/site-packages/trackintel/io/file.py:5: UserWarning: Shapely 2.0 is installed, but because PyGEOS is also installed, GeoPandas will still use PyGEOS by default for now. To force to use and test Shapely 2.0, you have to set the environment variable USE_PYGEOS=0. You can do this before starting the Python process, or in your code before importing geopandas:
import os os.environ['USE_PYGEOS'] = '0' import geopandas
In a future release, GeoPandas will switch to using Shapely by default. If you are using PyGEOS directly (calling PyGEOS functions on geometries from GeoPandas), this will then stop working and you are encouraged to migrate from PyGEOS to Shapely 2.0 (https://shapely.readthedocs.io/en/latest/migration_pygeos.html). import geopandas as gpd 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 182/182 [00:52<00:00, 3.45it/s] /home/adonnini1/anaconda3/lib/python3.9/site-packages/trackintel/preprocessing/positionfixes.py:111: UserWarning: 146900 duplicates were dropped from your positionfixes. Dropping duplicates is recommended but can be prevented using the 'exclude_duplicate_pfs' flag. warnings.warn(warn_str) 7%|███████████▏ | 12/182 [00:01<00:19, 8.87it/s]/home/adonnini1/anaconda3/lib/python3.9/site-packages/geopandas/_compat.py:123: UserWarning: The Shapely GEOS version (3.11.1-CAPI-1.17.1) is incompatible with the GEOS version PyGEOS was compiled with (3.10.4-CAPI-1.16.2). Conversions between both will be slow. warnings.warn( /home/adonnini1/anaconda3/lib/python3.9/site-packages/trackintel/io/file.py:5: UserWarning: Shapely 2.0 is installed, but because PyGEOS is also installed, GeoPandas will still use PyGEOS by default for now. To force to use and test Shapely 2.0, you have to set the environment variable USE_PYGEOS=0. You can do this before starting the Python process, or in your code before importing geopandas:
import os os.environ['USE_PYGEOS'] = '0' import geopandas
In a future release, GeoPandas will switch to using Shapely by default. If you are using PyGEOS directly (calling PyGEOS functions on geometries from GeoPandas), this will then stop working and you are encouraged to migrate from PyGEOS to Shapely 2.0 (https://shapely.readthedocs.io/en/latest/migration_pygeos.html). import geopandas as gpd /home/adonnini1/anaconda3/lib/python3.9/site-packages/geopandas/_compat.py:123: UserWarning: The Shapely GEOS version (3.11.1-CAPI-1.17.1) is incompatible with the GEOS version PyGEOS was compiled with (3.10.4-CAPI-1.16.2). Conversions between both will be slow. warnings.warn( /home/adonnini1/anaconda3/lib/python3.9/site-packages/trackintel/io/file.py:5: UserWarning: Shapely 2.0 is installed, but because PyGEOS is also installed, GeoPandas will still use PyGEOS by default for now. To force to use and test Shapely 2.0, you have to set the environment variable USE_PYGEOS=0. You can do this before starting the Python process, or in your code before importing geopandas:
import os os.environ['USE_PYGEOS'] = '0' import geopandas
In a future release, GeoPandas will switch to using Shapely by default. If you are using PyGEOS directly (calling PyGEOS functions on geometries from GeoPandas), this will then stop working and you are encouraged to migrate from PyGEOS to Shapely 2.0 (https://shapely.readthedocs.io/en/latest/migration_pygeos.html). import geopandas as gpd /home/adonnini1/anaconda3/lib/python3.9/site-packages/geopandas/_compat.py:123: UserWarning: The Shapely GEOS version (3.11.1-CAPI-1.17.1) is incompatible with the GEOS version PyGEOS was compiled with (3.10.4-CAPI-1.16.2). Conversions between both will be slow. warnings.warn( /home/adonnini1/anaconda3/lib/python3.9/site-packages/trackintel/io/file.py:5: UserWarning: Shapely 2.0 is installed, but because PyGEOS is also installed, GeoPandas will still use PyGEOS by default for now. To force to use and test Shapely 2.0, you have to set the environment variable USE_PYGEOS=0. You can do this before starting the Python process, or in your code before importing geopandas:
import os os.environ['USE_PYGEOS'] = '0' import geopandas
In a future release, GeoPandas will switch to using Shapely by default. If you are using PyGEOS directly (calling PyGEOS functions on geometries from GeoPandas), this will then stop working and you are encouraged to migrate from PyGEOS to Shapely 2.0 (https://shapely.readthedocs.io/en/latest/migration_pygeos.html). import geopandas as gpd /home/adonnini1/anaconda3/lib/python3.9/site-packages/geopandas/_compat.py:123: UserWarning: The Shapely GEOS version (3.11.1-CAPI-1.17.1) is incompatible with the GEOS version PyGEOS was compiled with (3.10.4-CAPI-1.16.2). Conversions between both will be slow. warnings.warn( /home/adonnini1/anaconda3/lib/python3.9/site-packages/trackintel/io/file.py:5: UserWarning: Shapely 2.0 is installed, but because PyGEOS is also installed, GeoPandas will still use PyGEOS by default for now. To force to use and test Shapely 2.0, you have to set the environment variable USE_PYGEOS=0. You can do this before starting the Python process, or in your code before importing geopandas:
import os os.environ['USE_PYGEOS'] = '0' import geopandas
In a future release, GeoPandas will switch to using Shapely by default. If you are using PyGEOS directly (calling PyGEOS functions on geometries from GeoPandas), this will then stop working and you are encouraged to migrate from PyGEOS to Shapely 2.0 (https://shapely.readthedocs.io/en/latest/migration_pygeos.html). import geopandas as gpd /home/adonnini1/anaconda3/lib/python3.9/site-packages/geopandas/_compat.py:123: UserWarning: The Shapely GEOS version (3.11.1-CAPI-1.17.1) is incompatible with the GEOS version PyGEOS was compiled with (3.10.4-CAPI-1.16.2). Conversions between both will be slow. warnings.warn( /home/adonnini1/anaconda3/lib/python3.9/site-packages/trackintel/io/file.py:5: UserWarning: Shapely 2.0 is installed, but because PyGEOS is also installed, GeoPandas will still use PyGEOS by default for now. To force to use and test Shapely 2.0, you have to set the environment variable USE_PYGEOS=0. You can do this before starting the Python process, or in your code before importing geopandas:
import os os.environ['USE_PYGEOS'] = '0' import geopandas
In a future release, GeoPandas will switch to using Shapely by default. If you are using PyGEOS directly (calling PyGEOS functions on geometries from GeoPandas), this will then stop working and you are encouraged to migrate from PyGEOS to Shapely 2.0 (https://shapely.readthedocs.io/en/latest/migration_pygeos.html). import geopandas as gpd /home/adonnini1/anaconda3/lib/python3.9/site-packages/geopandas/_compat.py:123: UserWarning: The Shapely GEOS version (3.11.1-CAPI-1.17.1) is incompatible with the GEOS version PyGEOS was compiled with (3.10.4-CAPI-1.16.2). Conversions between both will be slow. warnings.warn( /home/adonnini1/anaconda3/lib/python3.9/site-packages/trackintel/io/file.py:5: UserWarning: Shapely 2.0 is installed, but because PyGEOS is also installed, GeoPandas will still use PyGEOS by default for now. To force to use and test Shapely 2.0, you have to set the environment variable USE_PYGEOS=0. You can do this before starting the Python process, or in your code before importing geopandas:
import os os.environ['USE_PYGEOS'] = '0' import geopandas
In a future release, GeoPandas will switch to using Shapely by default. If you are using PyGEOS directly (calling PyGEOS functions on geometries from GeoPandas), this will then stop working and you are encouraged to migrate from PyGEOS to Shapely 2.0 (https://shapely.readthedocs.io/en/latest/migration_pygeos.html). import geopandas as gpd /home/adonnini1/anaconda3/lib/python3.9/site-packages/geopandas/_compat.py:123: UserWarning: The Shapely GEOS version (3.11.1-CAPI-1.17.1) is incompatible with the GEOS version PyGEOS was compiled with (3.10.4-CAPI-1.16.2). Conversions between both will be slow. warnings.warn( /home/adonnini1/anaconda3/lib/python3.9/site-packages/trackintel/io/file.py:5: UserWarning: Shapely 2.0 is installed, but because PyGEOS is also installed, GeoPandas will still use PyGEOS by default for now. To force to use and test Shapely 2.0, you have to set the environment variable USE_PYGEOS=0. You can do this before starting the Python process, or in your code before importing geopandas:
import os os.environ['USE_PYGEOS'] = '0' import geopandas
In a future release, GeoPandas will switch to using Shapely by default. If you are using PyGEOS directly (calling PyGEOS functions on geometries from GeoPandas), this will then stop working and you are encouraged to migrate from PyGEOS to Shapely 2.0 (https://shapely.readthedocs.io/en/latest/migration_pygeos.html). import geopandas as gpd /home/adonnini1/anaconda3/lib/python3.9/site-packages/geopandas/_compat.py:123: UserWarning: The Shapely GEOS version (3.11.1-CAPI-1.17.1) is incompatible with the GEOS version PyGEOS was compiled with (3.10.4-CAPI-1.16.2). Conversions between both will be slow. warnings.warn( /home/adonnini1/anaconda3/lib/python3.9/site-packages/trackintel/io/file.py:5: UserWarning: Shapely 2.0 is installed, but because PyGEOS is also installed, GeoPandas will still use PyGEOS by default for now. To force to use and test Shapely 2.0, you have to set the environment variable USE_PYGEOS=0. You can do this before starting the Python process, or in your code before importing geopandas:
import os os.environ['USE_PYGEOS'] = '0' import geopandas
In a future release, GeoPandas will switch to using Shapely by default. If you are using PyGEOS directly (calling PyGEOS functions on geometries from GeoPandas), this will then stop working and you are encouraged to migrate from PyGEOS to Shapely 2.0 (https://shapely.readthedocs.io/en/latest/migration_pygeos.html). import geopandas as gpd /home/adonnini1/anaconda3/lib/python3.9/site-packages/geopandas/_compat.py:123: UserWarning: The Shapely GEOS version (3.11.1-CAPI-1.17.1) is incompatible with the GEOS version PyGEOS was compiled with (3.10.4-CAPI-1.16.2). Conversions between both will be slow. warnings.warn( /home/adonnini1/anaconda3/lib/python3.9/site-packages/trackintel/io/file.py:5: UserWarning: Shapely 2.0 is installed, but because PyGEOS is also installed, GeoPandas will still use PyGEOS by default for now. To force to use and test Shapely 2.0, you have to set the environment variable USE_PYGEOS=0. You can do this before starting the Python process, or in your code before importing geopandas:
import os os.environ['USE_PYGEOS'] = '0' import geopandas
In a future release, GeoPandas will switch to using Shapely by default. If you are using PyGEOS directly (calling PyGEOS functions on geometries from GeoPandas), this will then stop working and you are encouraged to migrate from PyGEOS to Shapely 2.0 (https://shapely.readthedocs.io/en/latest/migration_pygeos.html). import geopandas as gpd /home/adonnini1/anaconda3/lib/python3.9/site-packages/geopandas/_compat.py:123: UserWarning: The Shapely GEOS version (3.11.1-CAPI-1.17.1) is incompatible with the GEOS version PyGEOS was compiled with (3.10.4-CAPI-1.16.2). Conversions between both will be slow. warnings.warn( /home/adonnini1/anaconda3/lib/python3.9/site-packages/trackintel/io/file.py:5: UserWarning: Shapely 2.0 is installed, but because PyGEOS is also installed, GeoPandas will still use PyGEOS by default for now. To force to use and test Shapely 2.0, you have to set the environment variable USE_PYGEOS=0. You can do this before starting the Python process, or in your code before importing geopandas:
import os os.environ['USE_PYGEOS'] = '0' import geopandas
In a future release, GeoPandas will switch to using Shapely by default. If you are using PyGEOS directly (calling PyGEOS functions on geometries from GeoPandas), this will then stop working and you are encouraged to migrate from PyGEOS to Shapely 2.0 (https://shapely.readthedocs.io/en/latest/migration_pygeos.html). import geopandas as gpd 13%|██████████████████████▎ | 24/182 [00:03<00:24, 6.37it/s]Traceback (most recent call last): File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction/prepro/geolife.py", line 151, in
get_npp_dataset(epsilon=args.epsilon, dataset=args.dataset, config=CONFIG)
File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction/prepro/geolife.py", line 25, in get_npp_dataset
pfs, sp = pfs.as_positionfixes.generate_staypoints(
File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/trackintel/model/positionfixes.py", line 84, in generate_staypoints
return ti.preprocessing.positionfixes.generate_staypoints(self._obj, *args, **kwargs)
File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/trackintel/preprocessing/positionfixes.py", line 128, in generate_staypoints
sp = applyParallel(
File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/trackintel/preprocessing/util.py", line 81, in applyParallel
df_ls = Parallel(n_jobs=n_jobs)(
File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/joblib/parallel.py", line 1056, in call
self.retrieve()
File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/joblib/parallel.py", line 935, in retrieve
self._output.extend(job.get(timeout=self.timeout))
File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/joblib/_parallel_backends.py", line 542, in wrap_future_result
return future.result(timeout=timeout)
File "/home/adonnini1/anaconda3/lib/python3.9/concurrent/futures/_base.py", line 446, in result
return self.get_result()
File "/home/adonnini1/anaconda3/lib/python3.9/concurrent/futures/_base.py", line 391, in get_result
raise self._exception
ValueError: unknown type timedelta64[ns]
26%|███████████████████████████████████████████▋ | 47/182 [00:04<00:12, 10.97it/s]
(base) adonnini1@actlnxlptp6:~/Development/ContextQSourceCode/NeuralNetworks/location-mode-prediction$