search

emacsen / changemonger

An OpenStreetMap change analyzer
GNU Affero General Public License v3.0
15 stars 4 forks source link

Handle extremely large changesets gracefully #21

Open ppawel opened 12 years ago

ppawel commented 12 years ago

Look at this bad boy:

http://www.openstreetmap.org/browse/changeset/13277695

OSC file is nearly 4MB!

I got HTTP timeout from Changemonger on this one. Perhaps there should be some limit of how many elements are processed to avoid timeouts?

samatjain commented 12 years ago

AFAIK, the timeouts are network-related—OpenStreetMap's API times out first while generating the OSC. It's not streamed, so Changemonger never begins to process anything.

I don't think OpenStreetMap's API can be easily modified to not do this for large changesets. Changemonger needs to look elsewhere for finding these large OSCs...

ppawel commented 12 years ago

I'm not using the public API in my branch - instead I'm using a local database. The timeout is definitely on Changemonger's side.

emacsen commented 12 years ago

I only see these timeout issues on the uwsgi hosted version. Can either of you confirm that you see them on private instances even against the public api?

ppawel commented 12 years ago

I do not dare to molest API with such large changesets :-) But in general my osmdbapi is faster than osmapi so I don't see how it would not time out in the same way.

samatjain commented 12 years ago

@ppawel Do you have a medium-sized changeset that we can run against the main API that causes changemonger to timeout? That is, a changeset that doesn't timeout in a Web browser, but does timeout with changemonger.

The changeset in the 1st comment times out in both my Web browser and changemonger.