Crunch-io / scrunch

Pythonic scripting library for cleaning data in Crunch
GNU Lesser General Public License v3.0
5 stars 7 forks source link

Export tabbook using multiple filters #424

Open jamesrkg opened 2 years ago

jamesrkg commented 2 years ago

See: https://github.com/Crunch-io/scrunch/blob/master/scrunch/subentity.py#L87-L178

Multitable.export and Multitable.export_tabbook both take a single filter, but the API supports multiple filters in combination (per docs here: https://crunch.io/api/reference/#post-/datasets/-dataset_id-/multitables/-multitable_id-/export/).

Therefore, we need to be able to export using multiple filters, for example:

mt = ds.multitables[mt_name]
mt.export(..., filters=[fltr1, fltr2])

I believe the only change needed is in Multitable.export_tabbook here:

        # add filter to multitable
        if filter:
            if isinstance(filter, Filter):
                payload['filter'] = [{'filter': filter.resource.self}]
            else:
                raise ValueError('filter param must be a Filter instance')

Where filter should support a list of filters, and be converted to a list if passed as a string originally, then [{'filter': filter.resource.self}] should instead be a list comprehension across that list such as: [{'filter': fltr.resource.self} for fltr in filter].

malecki commented 2 years ago

That differs from everywhere else in the API where filters=[A, B, C] means A & B & C.

We can think about how to do this with filters, BUT as an alternative, what do you think of another param such as filter_elements that took one variable and iterated through its categories or subvariables as if they were a sequence of filters?

jamesrkg commented 2 years ago

I probably could have been more explicit - my goal is to export with the intersection of the filter conditions given (as you say where filters=[A, B, C] means A & B & C). So the intention is that filters=[A, B, C] would only give you one export. I'm basically trying to mimic what we can do in the UI where we can apply multiple filters with a +.