search

jsvine / waybackpack

Download the entire Wayback Machine archive for a given URL.
MIT License
2.8k stars 189 forks source link

support multiple values for collapse #49

Closed shijialee closed 1 year ago

shijialee commented 2 years ago

per the doc "To use collapsing, add one or more collapse=field or collapse=field:N where N is the first N characters of field to test"

jsvine commented 2 years ago

Hi @shijialee, and interesting PR! It seems that, in order for multiple-collapse to be fully supported, further adjustments would need to be made to handle multiple values, such as here:

https://github.com/jsvine/waybackpack/blob/b6e8ec790d86d5fe3b65d358b651693195247678/waybackpack/cdx.py#L7-L22

shijialee commented 1 year ago

Completely forgot about this PR. Sorry about that.

Since argparser's append stores value as list, no need to change here for multiple values.

requests doc has the following sample code:

payload = {'key1': 'value1', 'key2': ['value2', 'value3']}
r = requests.get('https://httpbin.org/get', params=payload)

essentially, for collapse, this will be payload = {'collapse': ['timestamp:10', 'digest']}

jsvine commented 1 year ago

Belatedly(!): Thanks, that makes sense. Merging.