Launching the app in it's current state with the full Sao Paulo dataset will max out my 16GB RAM. This problem is entirely solved by limiting the data from around 20-50 stop_times per trip, to 2. The memory usage then drops to around 1GB.
For now, simply limit the number of routes (this is crude and only works well for SP, since there is 1 agency and always only 2 trips) so I can focus on making progress with features, and then revisit when I will understand Druid better, have a clearer list of architecture requirements necessary for features, can share a reasonably complete example with the Druid community to make it easier to ask for suggestions for ways to handle the large data and performance.
Possible solutions
Pagination could work but wouldn't be straight forward to know what number to group the routes by since different routes can have vastly different amounts of trips and stop_times.
A better solution might to not nest stop_times in trips in routes in agencies and just have everything eg in separate columns (but probably still all in the same column), so only one eg trips list can be viewed at a time (similar to how the stop works). This would also make it easier to achieve the goals of having sticky headers. This is likely to be the best solution and could just go ahead with this now but means we need to spend time building the functionality to update the AppState during runtime, so will postpone for the time being.
This is a catch all for all performance issues.
Launching the app in it's current state with the full Sao Paulo dataset will max out my 16GB RAM. This problem is entirely solved by limiting the data from around 20-50 stop_times per trip, to 2. The memory usage then drops to around 1GB.
For now, simply limit the number of routes (this is crude and only works well for SP, since there is 1 agency and always only 2 trips) so I can focus on making progress with features, and then revisit when I will understand Druid better, have a clearer list of architecture requirements necessary for features, can share a reasonably complete example with the Druid community to make it easier to ask for suggestions for ways to handle the large data and performance.
Possible solutions
Pagination could work but wouldn't be straight forward to know what number to group the routes by since different routes can have vastly different amounts of trips and stop_times.
A better solution might to not nest stop_times in trips in routes in agencies and just have everything eg in separate columns (but probably still all in the same column), so only one eg trips list can be viewed at a time (similar to how the stop works). This would also make it easier to achieve the goals of having sticky headers. This is likely to be the best solution and could just go ahead with this now but means we need to spend time building the functionality to update the AppState during runtime, so will postpone for the time being.