2-way data access solution: feedback

anastassiavybornova commented 5 days ago

@anerv

hey! so i've added the code and instructions for the 2-way solution as discussed: either a. the user refers to the already simplified and preprocessed data for all-of-DK (where data is publicly available), OR b. they refer to more recent data by doing an automated (or manual, if automated fails, incl. for cases where data is NOT publicly available) download and then explicit simplification.

to check the instructions for this new 2-way solution, start here: https://github.com/anastassiavybornova/bike-node-planner-data-denmark?tab=readme-ov-file#want-to-use-more-recent-data

and then follow the links.

happy about any feedback from your side on

does it make sense from a technical perspective
is it intuitive (or well explained) enough from a user perspective
more thoughts

also, feel free to update the documentation and/or code yourself if you think something should be changed! <3

anerv commented 4 days ago

Automatic download works, but I think it would be helpful to add some numbering to the instructions (at least I initially missed that the simplification step also had to be done when using the automatic option, which causes problem when you move on to step 4. So the structure could be:

**1. Download data 1.a Automatic 1.b. Manual

Preprocess** ...

After running the download (either automated or manual) + simplification (and getting a success statement for both), I get an error when moving on to the data generation:

Traceback (most recent call last):
  File "pyogrio/_io.pyx", line 1947, in pyogrio._io.ogr_create
  File "pyogrio/_err.pyx", line 183, in pyogrio._err.exc_wrap_pointer
pyogrio._err.CPLE_OpenFailedError: sqlite3_open(../input-for-bike-node-planner/network/raw/edges.gpkg) failed: unable to open database file

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/anev/Library/CloudStorage/Dropbox/ITU/repositories/bike-node-planner-data-denmark/scripts/generate-input.py", line 111, in <module>
    utils.get_edges_and_nodes(
  File "/Users/anev/Library/CloudStorage/Dropbox/ITU/repositories/bike-node-planner-data-denmark/src/utils.py", line 632, in get_edges_and_nodes
    edges_raw.to_file(
  File "/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9/site-packages/geopandas/geodataframe.py", line 1536, in to_file
    _to_file(self, filename, driver, schema, index, **kwargs)
  File "/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9/site-packages/geopandas/io/file.py", line 686, in _to_file
    _to_file_pyogrio(df, filename, driver, schema, crs, mode, metadata, **kwargs)
  File "/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9/site-packages/geopandas/io/file.py", line 748, in _to_file_pyogrio
    pyogrio.write_dataframe(df, filename, driver=driver, metadata=metadata, **kwargs)
  File "/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9/site-packages/pyogrio/geopandas.py", line 662, in write_dataframe
    write(
  File "/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9/site-packages/pyogrio/raw.py", line 723, in write
    ogr_write(
  File "pyogrio/_io.pyx", line 2307, in pyogrio._io.ogr_write
  File "pyogrio/_io.pyx", line 2135, in pyogrio._io.create_ogr_dataset_layer
  File "pyogrio/_io.pyx", line 1956, in pyogrio._io.ogr_create
pyogrio.errors.DataSourceError: sqlite3_open(../input-for-bike-node-planner/network/raw/edges.gpkg) failed: unable to open database file

It is correct that there is no such file, but I can't figure out in which step it should be generated?

anerv commented 4 days ago

For the manual download:

will the users only have access to the data for their municipality or can everyone with a login access the full network? Only asking because the simplification for the entire network would take a very long time, based on how long it is taking only for the publicly available data. In either case it might be nice to add a warning that the simplification step will take a little while?
When running the simplification using manually downloaded data I get a warning:

(knupu) anev@mac623123 bike-node-planner-data-denmark % python scripts/recent-data-simplify.py Subfolders created... Data folder cleaned... Edges read in and preprocessed. Simplifying network... /opt/miniconda3/envs/knupu/lib/python3.12/site-packages/sgeop/geometry.py:256: UserWarning: Could not create a connection as it would lead outside of the artifact. additions, splits = snap_to_targets(

anerv commented 4 days ago

Final comment - might be nice to remind users to deactivate the knupu environment after the simplification step and before running the rest of the data generation? I get errors when trying to run the main bash script with the knupu environment active, even though the command points to the QGIS python installation. I know that the bash script is not intended to be run from within the knupu environment, but it is easy to forget to deactivate it 😇

anerv commented 4 days ago

I've made a pull request with the edits I've suggested (added A/B to the automatic/manual options, warning that simplification can take a while, and reminder to deactivate knupu env)

anerv commented 4 days ago

Figured out the error with the edges in the raw folder - the sub-folders were missing :) So everything is taken care of with the two pull requests, expect the "UserWarning: Could not create a connection" problem

anastassiavybornova commented 3 days ago

hey! @anerv huge thanks for all of the above! the user warning comes from sgeop and i think we can just leave it there for now, so closing this issue <3

anastassiavybornova / bike-node-planner-data-denmark

2-way data access solution: feedback #37