Gpx export memory issue

[kojoty]

from oc.pl forum:

Post autor: Kipcior » poniedziałek, 8 maja 2017, 08:47 Żeby nie zakładać nowego tematu. Dzisiaj mam problem z eksportem wyników wyszukiwania do gpx. Jak zawsze wybrałem kryteria: Polska, wielkopolskie, reszta bez wykluczeń. Wyrzuciło mi liste ponad 3800 skrzynek. Po kliknięciu eksport do gpx wszystkich wyników wyszukiwania mieliło mieliło by po jakims czasie pokazac info: Firefox nie może odnaleźć pliku https://opencaching.pl/ocplgpx9909973.g ... =max&zip=1. Próbowałem kilka razy, zmieniając kryteria. Zawsze ten sam rezultat.

error example in log: [Mon May 08 09:38:35.706903 2017] [:error] [pid 28018] [client 81.15.211.114:5293] PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 100708352 bytes) in /srv/ocpl/lib/search.gpxgc.inc.php on line 565, referer: https://opencaching.pl/search.php?searchto=searchbyname&showresult=1&expert=0&output=HTML&sort=bydistance&f_inactive=0&f_ignored=0&f_userfound=0&f_userowner=0&f_watched=0&f_geokret=0&country=PL&region=PL41&cachetype=1111111110&cache_attribs=&cache_attribs_not=&cachesize_1=1&cachesize_2=1&cachesize_3=1&cachesize_4=1&cachesize_5=1&cachesize_6=1&cachesize_7=1&cachesize_8=%7Bcachesize_8%7D&cachevote_1=-3&cachevote_2=3.000&cachenovote=1&cachedifficulty_1=1&cachedifficulty_2=5&cacheterrain_1=1&cacheterrain_2=5&cacherating=0&cachename=%25&gpxLogLimit=100&cachename=

[andrixnet]

The query does return 3865 caches... and there are 100 logs requested for each cache... It is concievable that 100MB memory limit can't handle the actual amount of data.

[kojoty]

yeah, and we need to find a way to deal with such request or prevent such requests...

[andrixnet]

Possible solutions:

• increase memory limit. (partial workaround, the problem persists, only for larger sets of data).
• set a limit of maximum number of caches to be exported. (Groundspeak pocket queries limit this to 500, IMO such a limit will be hated).
• set a limit of maximum number of logs for each cache. (GSAK uses 30, GCTour GreaseMonkey script can use up to 100, but larger numbers really affect it's performance)
• change implementation such that no everything is held in memory, while generating. Since ob_flush and related core PHP functions are used to handle a way of directly outputting data or zip it, perhaps a way can be devised to flush the buffers every X caches such that memory limit problem is not encountered. (some of the previous implementations used user defined variables to simulate the behaviour of ob_flush)

[deg-pl]

From FB profile of RT: User > RT:

Cześć od dwóch dni nie mogę pobrać pliku gpx z sekcji "Eksport wszystkich skrzynek". Przez jakiś czas kręci się kółeczko ale się nie otwiera pole "Rozpoczęto pobieranie pliku". Potem otwiera się czysta strona w przeglądarce i koniec. Wcześniej wszystko działało bez zarzutu.

deg > user:

Hej - deg z tej strony Wczoraj w logach zauważyłem próby pobrania ogromnej ilości skrzynek - zakończone błędem. Być może to właśnie Twoje próby. Przed chwilą sprawdziłem dla mojej okolicy i bez problemu pobiera GPXa. Napisz proszę jakie filtry wyszukiwania stosujesz i ile skrzynek jest do pobrania.

user > RT:

Próbowałem pobrać skrzynki w promieniu 100km bez własnych i znalezionych. Łączna liczba skrzynek to 3500. Wcześniej nie było z tym problemu. Plik miał zwykle ok. 110MB. Pojedynczego GPX pobieram - problem jest przy pobieraniu grupy GPXów. a strona, która mi się pojawia na biało ma adres: https://opencaching.pl/ocplgpx9975037.gpx?startat=0&count=max&zip=1

From server logs: [Wed Sep 20 12:10:31.388279 2017] [:error] [pid 27521] [client 89.70.135.77:52676] PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 125579264 bytes) in /srv/ocpl/lib/search.gpxgc.inc.php on line 546, referer: https://opencaching.pl/search.php?searchto=searchbydistance&showresult=1&expert=0&output=HTML&sort=bydistance&f_userowner=1&f_userfound=1&f_inactive=1&latNS=N&lat_h=52&lat_min=07.550&lonEW=E&lon_h=21&lon_min=04.000&distance=100&unit=km

[deg-pl]

This is not good solution. I've added downloadMemorylimit and tried to export big (28765 caches) GPX. Server stoped to respond for many minutes. After over 10 minutes I restarted apache. There was no error in error.log.

[following5]

Have tested again after the rollback: The server is also blocked with the old implementation, so that's an old problem. It just gets worse by allowing even bigger downloads.

[following5]

This is a session handling problem. By default you cannot open a second page on the website as long another page is still loading in your web browser. Other users are not blocked.

A call to session_write_close() solves the problem.

[deg-pl]

Good news! Thanks for checking.

Have you any idea how to shorten time (resources?) needed to create large GPX?

[following5]

After some quick profiling, I see three options for optimization:

• Some SQL optimizations are possible. This is easy, but will not make it much faster. I guess it can save 10% of time.

• Get rid off all the string replacements and do it with a dynamic template like Okapi. I guess this will save about 30% of time, but it is much work.

• Offer an alternative download without personal information (ordering by distance from home location, modified coordinates, personal notes). This would allow to pre-generate GPX by some cronjob, which will make the GPX creation VERY fast. I guess 10-100 times faster.

[following5]

These figures came from testing on the develsite with mostly cached data. But now I did a 1500 caches GPX download from Opencaching.pl twice, and the second GPX was created MUCH faster. This points to the SQL queries as bottleneck.

[following5]

My fix - as proposed in #1252 and #1266 - would have immediately solved this issue, whatever maximum number of caches would be allowed to limit the download size and time.

But the discussion now is escalating in a way (see #1266 and #1273) which makes it too expensive for me to follow. Surely the GPX download can be optimized, but that's much work, and I wonder if it will ever be done.

I will retain my branch with the proposed fix for some weeks, for the case someone wants to finish it this way.