search

tcgoetz / GarminDB

Download and parse data from Garmin Connect or a Garmin watch, FitBit CSV, and MS Health CSV files into and analyze data in Sqlite serverless databases with Jupyter notebooks.
GNU General Public License v2.0
1.11k stars 137 forks source link

Processing FIT data takes 45mins #62

Closed giterest closed 4 years ago

giterest commented 4 years ago

Hi there, first off I'd like to shout out a big "Thank You!" for providing and maintaining "GarminDB". Everything works fine for me, but at the step "Processing FIT data from ..." - the "make" command is always processing all files there (6669 in my case) and that step takes almost 45mins. Would it be possible to only process NEW data?

Question: as far as I understood, the "make" command should be run, let's say weekly, in order to keep data up-to-date?

KR!

tcgoetz commented 4 years ago

make should only process new files. There have been bugs previously where the code that selects "new" files was returning all files.

Can you use the bug report script and attach the zip file it makes? That will give me some of the info I need to figure out why it's selecting all files for you.

To be clear, it's been more than 24 hours since you initially downloaded data? The data files won't be considered "old" until they have been around more than 24 hours. Can you also check the files date on disk?

giterest commented 4 years ago

Hi Tom, thanks for your reply! I guess there was something wrong with my VM - set up a new environment and will test again. Give me 48 hours so I can provide you /w updates.

Have a nice one!

giterest commented 4 years ago

Hi Tom, everything seems to be fine. Waited now for 48hours and GarminDB only processes "delta" data between last pull and now. Last question - if GarminConnectConfig.json has been intially set to 01/01/2019, can I force it to rebuild everything back to 2013?

tcgoetz commented 4 years ago

Yes, but there isn't a make command for that. You should look at the command the make command generates. Try changing the date and running:

/usr/local/bin/python3 garmin.py --all --download

giterest commented 4 years ago

Hi Tom, as I'm always waiting for 48 hours to perform makes/direct calls it takes a bit to answer.

So these are my settings;

        "weight_start_date"             : "01/01/2013",
        "sleep_start_date"              : "01/01/2013",
        "rhr_start_date"                : "01/01/2013",
        "monitoring_start_date"         : "01/01/2013",
        "download_days"                 : 31,
        "download_latest_activities"    : 10,
        "download_all_activities"       : 1000,
        "download_days_overlap"         : 3

resulting in getting activities, but no monitoring-data, whatsoever. I deleted the whole HealthData/ directory, but it didn't make any difference.

tcgoetz commented 4 years ago

Please attach the log file to this ticket.