search

arx-deidentifier / arx

ARX is a comprehensive open source data anonymization tool aiming to provide scalability and usability. It supports various anonymization techniques, methods for analyzing data quality and re-identification risks and it supports well-known privacy models, such as k-anonymity, l-diversity, t-closeness and differential privacy.
http://arx.deidentifier.org/
Apache License 2.0
618 stars 213 forks source link

[FEATURE/BUG] Enable definition of traning and validation set for privacy-preserving machine learning #377

Open prasser opened 2 years ago

prasser commented 2 years ago

Is your feature request related to a problem? Please describe.

The privacy-preserving machine learning framwork in ARX uses k-fold cross-validation to quantify the performance of privacy-preserving models. This can lead to misleading estimates as training and validation sets both influence the optimization process performed during anonymization.

Describe the solution you'd like As an alternative it would be good to enable users to specify a training and a validation set in such a way that only the training set influences the anonymization process. This can easily be done in ARX by using the "research subset" feature, which allows selecting a subset of the records in a dataset that are then anonymized. What would needed to be added is a feature that allows to specify that machine learning performance is determined based on the set of records that is not included in the research subset.

prasser commented 2 years ago

@srcds or @idhamari might want to take a look at this.

idhamari commented 2 years ago

@prasser @srcds sounds interesting, I will have a look at this next week.

idhamari commented 2 years ago

@prasser

When a user select random records by clicking on "Select Randomly", a view output does not reflect that e.g. by selecting 0.80:

    System.out.println("this.model.getOutput().getNumRows()                   : " + this.model.getOutput().getNumRows());
    System.out.println("this.model.getOutput().getView().getNumRows()         : " + this.model.getOutput().getView().getNumRows());
    System.out.println("this.model.getInputConfig().getResearchSubset.size()  : " + this.model.getInputConfig().getResearchSubset().size());

I get the output: 

         this.model.getOutput().getNumRows()                          : 30162
         this.model.getOutput().getView().getNumRows()          : 30162
         this.model.getInputConfig().getResearchSubset.size()  : 24070

I am using the view output to get the training and the testing subset which makes a problem.

prasser commented 2 years ago

This is OK and the expected behaviour.

prasser commented 2 years ago

An implementation of this is provided in the following branch: feature-training-test

Might still need a little bit of polishing, though.

jenno-verdonck commented 1 year ago

I was wondering if this feature can already be used it it's current state?

prasser commented 1 year ago

Yes, should work. We would be happy to receive feedback.

prasser commented 1 year ago

To be clear. The feature lives here in this branch:

https://github.com/arx-deidentifier/arx/tree/feature-training-test