search

conveyal / analyst-server

DEPRECATED; see https://github.com/conveyal/analysis-ui
MIT License
37 stars 6 forks source link

Spark caches entire upload in memory #253

Open abyrd opened 8 years ago

abyrd commented 8 years ago

Spark's spark.webserver.JettyHandler copies the entire input with IOUtils.toByteArray(super.getInputStream());

This doesn't work so well when uploading a 1GB+ PBF and GTFS bundle. The byte buffer expands when it gets full, exhausting all my heap.