search

peterwittek / happycowler

Crawl the HappyCow database to GPX or KML files for offline use
GNU General Public License v3.0
13 stars 4 forks source link

Show currently processed entry name #1

Closed thomasjfox closed 8 years ago

thomasjfox commented 8 years ago

Useful if parsing big cities like Berlin.

Thanks for happycowler, it was very neat to convert the resulting GPX files to a Garmin POI database.

peterwittek commented 8 years ago

This is really cool, thanks!

peterwittek commented 8 years ago

I made some changes to include a percentage of the progress. Release 0.2.3 now has this feature.

thomasjfox commented 8 years ago

Hi Peter,

Am Samstag, 17. September 2016, 07:33:48 schrieb Peter Wittek:

I made some changes to include a percentage of the progress. Release 0.2.3 now has this feature.

looking good!

Funny you also fixed the pagination / recursive call issue. I wanted to submit that one, too, but when I searched for the code change in the diff output after returning home, I was unable to spot it again.

Regarding the progress bar: Is it on purpose it shows 0.00% percent all the time? Is the progress by per full page?

Something related: May be the script could be splitted into two parts. One part that does the download and one part that does the parsing. That way one could hack around the parser without querying the server all the time. Also small unit tests for the parser would be possible. That would also make the progress bar more accurate.

Just ideas though, I'm quite happy with the current state :)

peterwittek commented 8 years ago

The progress should be overall: the total number of places appear in the top part of the results. The percentage shows correctly for Barcelona:

happycowler https://www.happycow.net/europe/spain/barcelona/ whatever.gpx

Could you open an issue on this with the city where it is broken?

I also open an issue on the splitting. Filewriters should go to separate files, and I will try to do something about separating crawling.

peterwittek commented 8 years ago

Modularization is now done and unit tests are included.

thomasjfox commented 8 years ago

Hi Peter,

Am Montag, 19. September 2016, 21:38:28 schrieb Peter Wittek:

The progress should be overall: the total number of places appear in the top part of the results. The percentage shows correctly for Barcelona:

happycowler https://www.happycow.net/europe/spain/barcelona/ whatever.gpx

Could you open an issue on this with the city where it is broken?

I also open an issue on the splitting. Filewriters should go to separate files, and I will try to do something about separating crawling.

is it possible to invoke the latest git HEAD code from the cmdline without installing it?

I tried this:

$ python3 happycowler https://www.happycow.net/europe/spain/barcelona/ whatever.gpx /usr/bin/python3: can't find 'main' module in 'happycowler

or this:

$ python3 -m happycowler /usr/bin/python3: No module named happycowler.main; 'happycowler' is a package and cannot be directly executed

I'm pretty sure there's an easy trick to this :)

btw: The invocation example in README.rst contains the cmd "happycow" instead of "happycowler".

Cheers, Thomas

peterwittek commented 8 years ago

This should work:

python3 scripts/happycowl https://www.happycow.net/europe/spain/barcelona/ whatever.gpx

The script was renamed to happycowl because now happycowler actually is a module.

thomasjfox commented 8 years ago

Am Montag, 10. Oktober 2016, 12:27:09 schrieb Peter Wittek:

This should work:

python3 scripts/happycowl https://www.happycow.net/europe/spain/barcelona/
whatever.gpx ```
The script was renamed to `happycowl` because now `happycowler` actually is
a module.

ah sorry, I didn't see the new script name because my "scripts" subdir is polluted with a lot of .gpx output files.

Back to the original question: The progress bar is working for me with the latest code. Issue solved :)

Cheers, Thomas