Combine trip files into single file?

RSGInc / micromobility_toolset

Micromobility Travel Modeling Toolkit

Other

3 stars 2 forks source link

Combine trip files into single file? #23

Open blakerosenthal opened 3 years ago

blakerosenthal commented 3 years ago

Currently we output a separate trip file indexed pzone, azone, trips for each segment in trips.yaml, which is kind of cluttery. Would it be better to have a trips.csv file with a header like pzone, azone, segment1, segment2, ...?

Pros:

less clutter
smaller config file (don't need to specify file names)

Cons:

slightly larger file size overall since segmented trip files will exclude zeros, but a unified file might include a lot of zeros for certain segments
possibly could require changes to post-process analysis scripts

@joshua-reynolds?

joshua-reynolds commented 3 years ago

If its an easy switch, I'd say go ahead and do it if we have the budget. Would be nice to only have one trip.csv file. My post processing function would be easy to modify for this.

I'm not too concerned about the file size since the individual are already large and can't be completely shown using excel.

I did some light data science work to examine how large the table would be. Based on the fake table I created, it seems like "trips.csv" would come in at around ~3.5gb. It wouldn't load at all in excel; not even partially lol. Pandas would basically required to do any kind of post processing, but I think that's fine.

blakerosenthal commented 3 years ago

Okay, great. Wanted to make sure I wouldn't break anything if I made that change. I have a hard time scanning the outputs folder sometimes with all those trip files, so combining them seemed convenient. I'll wrap that into my next round of fixes.