search

Safecast / safecastapi

The app that powers api.safecast.org
44 stars 25 forks source link

Performance - upload process slow (bulk upload function missing) #53

Closed nokton closed 11 years ago

nokton commented 11 years ago

Sometime takes hours. Overall response slow.

paulca commented 11 years ago

I think we need to define a new upload process, where you can use a third party for uploads. Maybe Dropbox?

nokton commented 11 years ago

Hi Paul, the upload of the file itself is "OK" - what takes time is the time to parse the file, etc.

Pieter

On Jan 13, 2013, at 2:00 PM, Paul Campbell notifications@github.com wrote:

I think we need to define a new upload process, where you can use a third party for uploads. Maybe Dropbox?

— Reply to this email directly or view it on GitHub.

thinrope commented 11 years ago

Map preview on the upload page often takes forever.... How is it generated? Every time I switch to the tab, it starts doing something (eating RAM/CPU) and showing "Loading map" progress bar, but when I switch to another tab it seems to halt... FF17 on linux... Currently working on https://api.safecast.org/en-US/bgeigie_imports/11974

paulca commented 11 years ago

I pushed a fix for this in 6963d94903890a4d7fec3a7ddcd70755027eed26.

Basically, we were converting NMEA to floating point lat/lng row by row in a single transaction in Ruby. Read: slow.

The real bug though was that for every file, the entire bgeigie_logs table (6.5m rows and counting) was being recomputed. Read: days and days of processing.

The fix, then is twofold. First, shift the processing computation to SQL for a massive processing speedup: https://github.com/Safecast/safecastapi/commit/6963d94903890a4d7fec3a7ddcd70755027eed26#L0R181

Second, compute the floating point lat/lng for the file while it's still in a temporary table, so we're only converting as many records as are in that file: https://github.com/Safecast/safecastapi/commit/6963d94903890a4d7fec3a7ddcd70755027eed26#L0R174

Result: massive speedup.

robouden commented 11 years ago

Paul

Nice, great progress... I will check it out soon.

regards rob

On Sun, Apr 28, 2013 at 6:43 PM, Paul Campbell notifications@github.comwrote:

I pushed a fix for this in 6963d94https://github.com/Safecast/safecastapi/commit/6963d94903890a4d7fec3a7ddcd70755027eed26 .

Basically, we were converting NMEA to floating point lat/lng row by row in a single transaction in Ruby. Read: slow.

The real bug though was that for every file, the entire bgeigie_logs table (6.5m rows and counting) was being recomputed. Read: days and days of processing.

The fix, then is twofold. First, shift the processing computation to SQL for a massive processing speedup: 6963d94#L0R181https://github.com/Safecast/safecastapi/commit/6963d94903890a4d7fec3a7ddcd70755027eed26#L0R181

Second, compute the floating point lat/lng for the file while it's still in a temporary table, so we're only converting as many records as are in that file: 6963d94#L0R174https://github.com/Safecast/safecastapi/commit/6963d94903890a4d7fec3a7ddcd70755027eed26#L0R174

Result: massive speedup.

— Reply to this email directly or view it on GitHubhttps://github.com/Safecast/safecastapi/issues/53#issuecomment-17131347 .

Regards, Rob Oudendijk Yuka Hayashi http://yr-design.biz http://oudendijk.biz http://about.me/robouden tel +81 80-22605966 Skype: robouden Facebook:roboudenhttp://on.fb.me/QeKw2P twitter:robouden http://bit.ly/RAiSTC

nokton commented 11 years ago

Much appreciated - this is massive progress!!

One more in this class is we have 200 drives stuck for some reason - how can we trigger a reprocessing of these.

Pieter

On 2013/04/28, at 6:19, Rob Oudendijk notifications@github.com wrote:

Paul

Nice, great progress... I will check it out soon.

regards rob

On Sun, Apr 28, 2013 at 6:43 PM, Paul Campbell notifications@github.comwrote:

I pushed a fix for this in 6963d94https://github.com/Safecast/safecastapi/commit/6963d94903890a4d7fec3a7ddcd70755027eed26 .

Basically, we were converting NMEA to floating point lat/lng row by row in a single transaction in Ruby. Read: slow.

The real bug though was that for every file, the entire bgeigie_logs table (6.5m rows and counting) was being recomputed. Read: days and days of processing.

The fix, then is twofold. First, shift the processing computation to SQL for a massive processing speedup: 6963d94#L0R181https://github.com/Safecast/safecastapi/commit/6963d94903890a4d7fec3a7ddcd70755027eed26#L0R181

Second, compute the floating point lat/lng for the file while it's still in a temporary table, so we're only converting as many records as are in that file: 6963d94#L0R174https://github.com/Safecast/safecastapi/commit/6963d94903890a4d7fec3a7ddcd70755027eed26#L0R174

Result: massive speedup.

\ Reply to this email directly or view it on GitHubhttps://github.com/Safecast/safecastapi/issues/53#issuecomment-17131347 .

Regards, Rob Oudendijk Yuka Hayashi http://yr-design.biz http://oudendijk.biz http://about.me/robouden tel +81 80-22605966 Skype: robouden Facebook:roboudenhttp://on.fb.me/QeKw2P twitter:robouden http://bit.ly/RAiSTC \ Reply to this email directly or view it on GitHub.

paulca commented 11 years ago

Where are the stuck drives?

On Sun, Apr 28, 2013 at 3:03 PM, Pieter Franken notifications@github.comwrote:

Much appreciated - this is massive progress!!

One more in this class is we have 200 drives stuck for some reason - how can we trigger a reprocessing of these.

Pieter

On 2013/04/28, at 6:19, Rob Oudendijk notifications@github.com wrote:

Paul

Nice, great progress... I will check it out soon.

regards rob

On Sun, Apr 28, 2013 at 6:43 PM, Paul Campbell notifications@github.comwrote:

I pushed a fix for this in 6963d94< https://github.com/Safecast/safecastapi/commit/6963d94903890a4d7fec3a7ddcd70755027eed26>

.

Basically, we were converting NMEA to floating point lat/lng row by row in a single transaction in Ruby. Read: slow.

The real bug though was that for every file, the entire bgeigie_logs table (6.5m rows and counting) was being recomputed. Read: days and days of processing.

The fix, then is twofold. First, shift the processing computation to SQL for a massive processing speedup: 6963d94#L0R181< https://github.com/Safecast/safecastapi/commit/6963d94903890a4d7fec3a7ddcd70755027eed26#L0R181>

Second, compute the floating point lat/lng for the file while it's still in a temporary table, so we're only converting as many records as are in that file: 6963d94#L0R174< https://github.com/Safecast/safecastapi/commit/6963d94903890a4d7fec3a7ddcd70755027eed26#L0R174>

Result: massive speedup.

\ Reply to this email directly or view it on GitHub< https://github.com/Safecast/safecastapi/issues/53#issuecomment-17131347> .

Regards, Rob Oudendijk Yuka Hayashi http://yr-design.biz http://oudendijk.biz http://about.me/robouden tel +81 80-22605966 Skype: robouden Facebook:robouden< http://on.fb.me/QeKw2P> twitter:robouden http://bit.ly/RAiSTC \ Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHubhttps://github.com/Safecast/safecastapi/issues/53#issuecomment-17140399 .

Paul Campbell paul@hypertiny.ie

web http://hypertiny.ie blog http://www.pabcas.com twitter http://www.twitter.com/paulca github http://www.github.com/paulca phone +353 87 914 8162