Closed joelis10 closed 1 month ago
i would guess reverse geocoding error? can you do(a bit of disk size heavy) photon with dawarich? do another vm,if i remember correctly,it was more cpu tied than memory(it was using around 2GB ram),disk size i recommend 200GB...there is the issue of getting those 200GB-download(almost 100GB of data ?(g)zipped?) to another disk and unpack to those 200GB,otherwise you need more...after starting it,it eats ~20% of 2 cores on my 7600k for few seconds during resolving(each point)
tldr:free service of geocoding api limit or down
edit:i use dawarich as secondary,i always have source data,so i can wipe and restart,import,start photon again(i use it for first time import only,i have around 1.1GB google export raw text points),there is start/continue geocoding in gui settings,i believe those errors are not a problem for you,but if you have doubts,you can do it same as me,and adding photon (there should be some basic info how to photon,maybe on one release,test on a few points and then do overnight mass import)
I'm sorry I have no idea what you mean by Photon (and really anything you just said)
photon is this: https://github.com/komoot/photon ,reverse geocoding is service where you give it api coordinates,it replies with name(city,street,maybe shop name),this service is cpu intensive,so limits are used,which you reached probably,or just bad time and it was down,you can bypass this with your own selfhosted,dawarich allows you to set your own geocoding service(photon),while it is still not completely selfhosted(you pull maps from openstreetmap provider),that could be sorted with another map proxy...
selfhosted means you are your own limit,if it does not work,it is your problem :)
I'm running into this same issue. 1.5M processed, 494k failed so far. Retries panel says:
[just now](http://localhost:3000/sidekiq/retries/1725941155.4189491-d77c8f8b7897bcd27e5d6f38) 0 [reverse_geocoding](http://localhost:3000/sidekiq/queues/reverse_geocoding) ReverseGeocodingJob
"Point", 775360
Geocoder::ResponseParseError: Geocoder::ResponseParseError
many times.
I don't understand @Korenchkin 's suggestion enough to implement it. Would you mind writing up a FAQ about what to do here?
I was going to say the same as @jonhnet - I'm sure there'd be a proper solution the dev(s) can share as this can't be the first time it's happened
Failed jobs != failed point imports. If you see your file on the Imports page, and it has some number of points against it, then it was successfully imported.
The other thing is reverse geocoding, which might fail due to ban from Nominatim or Photon instance misconfiguration. In the original problem description from @joelis10 I see exactly a problem with an invalid response from reverse geocoding service, but not with an import failure. In any case, it has nothing to do with Dawarich, but with reverse geocoding service, be it Nominatim or your own Photon API instance.
Following this guide should solve this problem: https://dawarich.app/docs/tutorials/reverse-geocoding#setting-up-your-own-reverse-geocoding-service
Kudos for providing proper logs though 👍
@Freika The linked guide describes but doesn't link to: https://tonsnoei.nl/en/post/2023/03/20/set-up-your-own-geocoder-api/ I'd send this suggestion as a PR, but I don't see where the tutorials live in the github tree. ...I'm off following the instructions at the linked tutorial now. Thank you!
@Freika The linked guide describes but doesn't link to
https://github.com/dawarich-app/site/blob/main/docs/tutorials/reverse-geocoding.md?plain=1#L49
Oh dear, it was there all along! Sorry about that. On my display the hyperlinks weren't rendered differently than ordinary text, so it didn't occur to me to click. Sorry about that!
@jonhnet that made me realise that light version of the website actually doesn't highlight links in any visible way. Will be fixed :)
Describe the bug Title, with proof (I may have been exaggerating a little):
Version 0.13.2, as that's what I started the import on a few days ago and obviously didn't want to update in the middle of an import.
To Reproduce Steps to reproduce the behavior:
Expected behavior All points successfully import without issue
Screenshots See above.
Logs sidekiq last 100 lines:
and main dawarich_app container:
note that I took these logs out of their containers a few seconds apart, so they might not 100% match each other Additional context My import is still going as I speak, but I wanted to see if this was a known issue/if it would fix itself once actually done instead of waiting another 24 hours or so for it to finish before I find out.