search

VolpeUSDOT / RDR-Public

Public Version of the Resilience and Disaster Recovery Tool Suite
Other
6 stars 2 forks source link

Data type Error with network CSV files #13

Closed nicholasfisher-rsg closed 1 week ago

nicholasfisher-rsg commented 3 weeks ago

Hi, I am attempting to do a customized RDR run using a travel model network for the links and nodes. I have formatted the network CSV files into the format needed for RDR, but seem to run into this error (see below) when running the tool, it appears to be related to data types in the network CSVs (based on the number of observations [0-10007] the issue is in my network links files where I have 10008 links).

Is this something I can fix on my end or is there a different solution required? I was able to run the quickstart and used those files and the user guide as the templates for my own inputs. As far as I can tell they are in the same format.

Error message:

08-21 17:00:18 RESULT AequilibraE network links table written to C:\RDR-Public\scenarios\s1_srtc\Data\generated_files\aeq_runs\base\s1\base02\matrix\Group02_baserun.csv 08-21 17:00:18 INFO GMNS_link table to be filled from C:\RDR-Public\scenarios\s1_srtc\Data\generated_files\aeq_runs\base\s1\base02\matrix\Group02_baserun.csv C:\Users\nicholas.fisher\Anaconda3\envs\RDR_env\Lib\site-packages\aequilibrae\project\network\network.py:337: FutureWarning: Downcasting object dtype arrays on .fillna, .ffill, .bfill is deprecated and will change in a future version. Call result.infer_objects(copy=False) instead. To opt-in to the future behavior, set pd.set_option('future.no_silent_downcasting', True) df = pd.read_sql(sql, conn).fillna(value=np.nan) C:\Users\nicholas.fisher\Anaconda3\envs\RDR_env\Lib\site-packages\aequilibrae\paths\graph.py:221: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[ 0 1 2 ... 10005 10006 10007]' has dtype incompatible with int32, please explicitly cast to a compatible dtype first. df.loc[:, "id"] = df.id.values.astype(self.integer_type) !!!!!!!!!!!!!!!!!!!! EXCEPTION RAISED !!!!!!!!!!!!!!!!!!!!!! Traceback (most recent call last): File "C:\RDR-Public\metamodel_py\Run_RDR.py", line 160, in main main(input_folder, output_folder, cfg, logger) File "C:\RDR-Public\metamodel_py\rdr_RunAE.py", line 82, in main rdr_AESingleRun.run_AESingleRun(run_params, input_folder, output_folder, cfg, logger) File "C:\RDR-Public\metamodel_py\rdr_AESingleRun.py", line 110, in run_AESingleRun run_aeq_base(run_params, base_run_folder, cfg, logger) File "C:\RDR-Public\metamodel_py\rdr_AERouteBase.py", line 49, in run_aeq_base project.network.build_graphs() File "C:\Users\nicholas.fisher\Anaconda3\envs\RDR_env\Lib\site-packages\aequilibrae\project\network\network.py", line 351, in build_graphs g.prepare_graph(centroids) File "C:\Users\nicholas.fisher\Anaconda3\envs\RDR_env\Lib\site-packages\aequilibrae\paths\graph.py", line 149, in prepare_graph self.build_compressed_graph() File "C:\Users\nicholas.fisher\Anaconda3\envs\RDR_env\Lib\site-packages\aequilibrae\paths\graph.py", line 153, in __build_compressed_graph build_compressed_graph(self) File "aequilibrae\paths\graph_building.pyx", line 130, in aequilibrae.paths.AoN.build_compressed_graph IndexError: index 28720 is out of bounds for axis 0 with size 28720 !!!!!!!!!!!!!!!!!!!! EXCEPTION RAISED !!!!!!!!!!!!!!!!!!!!!!

kzhang81 commented 3 weeks ago

Hi @nicholasfisher-rsg thanks for reaching out and documenting the issue!

Would you be able to share your network node and link CSV input files? A copy of the specific config file you're using may be helpful as well. We will look into the issue--feel free to share in this GH issue, or email us at RDR-Team@dot.gov.

Some initial thoughts without having had a chance to look at the specific file causing the issue:

Hopefully one of those will help resolve things; otherwise we can offer more support once we've taken a look at the specific input files. Thanks!

nicholasfisher-rsg commented 3 weeks ago

Thanks for the reply. I've followed your suggested instructions and am still running into the same error. I did not see any trailing decimals when looking at the network files in a text editor so I'm not sure where the data type issue is coming from. I'm also not able to generate the metamodel input file due to the same error.

I've attached both my links and nodes files along with my config file (this is attached as a .txt file). The links represent unique travel model links and the nodes represent unique non-centroid travel model nodes. Thanks again.

baseyear.csv node.csv s1_config.txt

nicholasfisher-rsg commented 2 weeks ago

Hi there, following up on this thread. Anything else I can provide to help assess the issue?

kzhang81 commented 2 weeks ago

Thanks for following up--we're planning to take a closer look at your files today and will reach back out! Those files should be sufficient for now, though if you have a trip table OMX you'd be able to share that could be useful.

kzhang81 commented 2 weeks ago

Hi @nicholasfisher-rsg -- thanks for your patience! I took a look at these network CSV files and identified a few steps that should fix the errors you're seeing:

  1. Your node CSV file needs to include centroid nodes -- these are locations where trip originate and end, and the number of centroid nodes should correspond to the number of rows (and number of columns) in your trip table OMX file. These centroid nodes should also have smaller node IDs than non-centroid nodes.
  2. I'd recommend renumbering your nodes to have consecutive node IDs starting from 1 -- you should feel free to keep the original node IDs in a separate column (e.g., orig_node_id) for record-keeping, but AequilibraE will work best with consecutive node IDs (e.g., 1 to 8821). Note that these will also need to be updated in your link CSV file.
  3. Finally, there are several links with zero free_flow_time -- AequilibraE will throw an error for these. When I changed them to 0.01, I was able to run AequilibraE successfully for your input files.

Hope that will resolve your issues! If you have any follow-on questions, please feel free to reach out again!

nicholasfisher-rsg commented 2 weeks ago

Can you clarify fix number 3 in your post? I do not see any links with a speed less than 20mph.

I have followed these steps and am still encountering the same error.

kzhang81 commented 2 weeks ago

Hi @nicholasfisher-rsg -- for number 3, there are zeroes in the free_flow_time column, not the speed column.

I recommend using the baseline network helper tool to start--that tool only requires the files you shared above, along with a trip table (and template SQLite database). To debug further, it'd be great to get a copy of your trip table and compare it to your node CSV file. Happy to look at them separately, or if you'd like to set up some time to chat, feel free to send an email to RDR-Team@dot.gov. Thanks!

nicholasfisher-rsg commented 2 weeks ago

Thanks for the clarification, Imisread your reply. I am having trouble finding the free_flow_time column, can you direct me to which file that is located in?

kzhang81 commented 2 weeks ago

No worries--that column should be in your baseyear.csv link CSV file.

nicholasfisher-rsg commented 2 weeks ago

I sent a message to the provided email to set up a call to troubleshoot this scenario but received a reply that it was undeliverable. I would still like to set something up to discuss this scenario at your earliest availability. Please advise on how I can get in contact with the RDR team, I can be reached at nicholas.fisher@rsginc.com. Thanks!

image

kzhang81 commented 2 weeks ago

Thanks for flagging that--we'll figure that out! And I will follow up with an email to you!

kzhang81 commented 1 week ago

Closing RDR-Public issue as conversation has shifted to email.