New feature 'exclude-filter'

chs8691 commented 3 years ago

Summary

Exclude activities from the download by a given json file with an array of activity IDs.

Use Case

The user wants the ability to exclude activities by its ID from the download. This has to be independent from the already downloaded activity files; activities have to be excluded, even if the activity file does not exist in the given download folder. In addition to that, the user expects a managed json file with the list of downloaded activities, which can be used as input for the exclude filter. The file should be stored in the download folder and has to be named "downloaded_ids.json".

Usage by examples:

1) Download the last 10 activities, add IDs to downloads/downloaded_ids.json

$ python gcexport.py -f tcx -d downloads  -s in -fp -c 10

2) Download just the 11th activity. Add its ID to downloads/downloaded_ids.json

$ python gcexport.py -f tcx -d downloads  -s in -fp -c 11 -ex downloads/downloaded_ids.json

3) Download the last 100 activities, skip the last 10, exclude activity '6540566061'

$ echo '{"ids": ["6540566061"]}' > ex.json
$ python gcexport.py -f tcx -d downloads  -s in -fp -c 100 -sa 10 -ex ex.json

Implementation details

The large processing loop is now split into two parts. First the data will be collected and the action for every activity will be determined (skip, exclude and download). The second part is the loop over this action list to output and download the files. With every successful download the json file 'downloaded_ids.json' will be extended with the activity ID (skipping doesn't do this). Beside this, there is just one change in the former behavior: the userstats will always be downloaded, not only for 'count all'. This is obligatory to get the activity IDs. It will be also downloaded, even if the exclude filter is not used. In my opinion, this is more transparent to the user, because the userstats.json is always up-to-date. On the other hand, there is at least one request more, but this should be acceptable at all.

One full example output

$ cat downloads/downloaded_ids.json
{"ids": ["6540566061", "6563511175"]}
$ python gcexport.py -f tcx -d downloads -s in -fp -c 4 -sa 2 -ex downloads/downloaded_ids.json
Welcome to Garmin Connect Exporter!
[WARNING] Output directory downloads already exists. Will skip already-downloaded files and append to the CSV file.
Connecting to Garmin Connect... Done.
Requesting Login ticket... Done. Ticket=ST-03005971-4rvXNVtz3YV34NHoXFAV-cas
Authenticating... Done.
Getting display name... Done. displayName=6b970a16-d759-4e7f-8b1e-e6c170e96ecd
Fetching user stats... Done.
Querying list of activities 1..4... Done.
Skipping   : Garmin Connect activity (1/4) [6574633315] 
Excluding  : Garmin Connect activity (2/4) [6563511175] 
Excluding  : Garmin Connect activity (3/4) [6540566061] 
Downloading: Garmin Connect activity (4/4) [6510862175] Gelnhausen Radfahren
        2021-03-29T06:55:58+02:00, 01:23:23, 29.279km
Done!

pe-st commented 3 years ago

Hello @chs8691

I like how you refactored the loop :-), so I stole your idea for the feature to support multisport activities which I wanted to do for a long time.

So I pushed two branches: one with the multisport support and a loop refactoring inspired by your PR (see https://github.com/pe-st/garmin-connect-export/tree/feature/multi-sport), but I split the refactoring over several commits to make it easier to see what changed.

The second branch https://github.com/pe-st/garmin-connect-export/tree/feature/exclude-filter adds your new feature on top of the first branch. I separated the creation of the action list from the other loop logic to have simpler methods...

I'll have to test the multisport feature still for a bit (and add doc etc) before I integrate it into the master. Afterwards I can merge your PR by replacing with the code from my second branch, if that is okay with you, so you would remain the author; alternatively you could merge/rebase your PR once the multisport feature is in master.

If you want I can also add you as co-author for the refactoring commits on the multisport branch.

Cheers!

chs8691 commented 3 years ago

Hi pe-st, Sounds good ... and goog luck for merging ;0) Thanks for your explanation. Chris

pe-st commented 3 years ago

Merged and released with 3.1.0

pe-st / garmin-connect-export