obrok
commented
6 years ago One way of dealing with this is to keep the model stored inside the cloak. This would mean that the analysts can only obtain predictions by sending new inputs to cloak. I'm not sure how useful would this be in practice though. More likely, analysts are accustomed to using their own tools, which are likely feature rich beyond what we can achieve with a reasonable effort.
Another option is to try and sanitize the obtained prediction model before sending it back. The purpose of the sanitization is to remove all the branches which can compromise an individual (or a small group). I'm not completely sure whether this resolves all the privacy issues though.
My immediate thought is that this is impossible without a significant effort put into understanding these models and maybe developing a privacy-specific prediction model. The prediction model "compresses" the knowledge about the domain, so it's difficult to impossible to tweak this compressed representation so that no privacy is leaked. Even if we only allow querying the model from outside that seems like very easily exploitable - you teach the model on only a small amount of data that you want to attack and then when you query it, it will just replay the exact values.
In the extension of this idea, the cloak would somehow figure out which fields should be premasked to increase the amount of the non-masked data returned. The benefit is then that analysts can issue their select from an arbitrary table (or join), get their data and use the result with any desired tool for any purpose, such as supervised or unsupervised ML. In other words, by improving the usefulness of the returned set, we can solve many problems.
We could also give the analyst a suite of functions that would allow them to figure out which fields need to be masked or not, leaving the final decision in their hands, but speeding up the process.
sebastian
sasa1977
cristianberneanu
This is a summary of my current thoughts on supporting machine learning through Aircloak. I should note that I'm not very familiar with ML, so I might be missing some important points. Feel free to correct me if I made some incorrect statements :-)
The overall goal is to improve ML scenarios on the data obtained through Aircloak. In this discussion I'll only focus on the supervised ML, since that's all I've looked into so far.
The high-level simplified story of the supervised learning is that the machine builds a prediction model based on some input dataset. In the dataset, we have one or more input fields, and we have one or more output fields. The machine crunches the dataset and spits out a model which can be used to predict the output values for any combination of input values. For example, in the input dataset, we could have a bunch of boolean fields such as
fever,headache, ..., and the output could be thediagnosisfield (e.g. flu, diabetes, ...).There are of course a bunch of algorithms for this, and there are various off-the-shelf solution. However, the problem with such solutions is that we need to fetch all the data through the Aircloak, and this could likely lead to everything being anonymized away. Considering the example above, let's say we have the table called
patientsand we doselect * from patients. If there are some highly unique fields in the table (for example SSN), we won't get any meaningful results to build a reliable prediction model.A simple fix can be done if an analyst is familiar with the data. The analyst could only select the relevant fields (such as symptom fields and the
diagnosisfield). However, in many cases this might not be a viable option. Even when it is possible to determine relevant input fields, we'd still lose some fidelity in the input set, because analysts need to select all the input fields at once. I'm currently not sure if this is necessarily a bad thing, because from the standpoint of the machine learning, we don't want outlier combinations to affect the model. So perhaps the fact that we're masking infrequent combinations is in fact a good thing.Either way it's worth considering whether we can somehow improve the reliability of the prediction model. I'll discuss a couple of ideas here.
Provide our own external Aircloak-aware ML implementations
In this proposal, we implement ML algorithms ourselves, trying to take advantage of how data is best retrieved from Aircloak. From what I can tell, in most supervised ML implementations, the algorithm is repeatedly trying to figure out which field leads to the "best" data split. For example, if we're trying to figure out the symptoms for a flu, the
feverfield might be a much better splitter thantoothache.In order to determine which field is a better splitter, we need to correlate each input field with the output field. Therefore, instead of doing
select fever, toothache, diagnosis, we could issue two selects:select fever, diagnosisandselect toothache, diagnosis, which might give us more data to work with.On the flip side, I see a bunch of challenges here:
So considering these issues, my current impression is that this is a quite huge undertaking, with a very uncertain outcome. While I have no doubt that we can make it work somehow, I'm worried about the usefulness of the produced solution. Given that Aircloak itself is slow, I have some serious concerns whether our solution which repeatedly issues a bunch of queries to the Aircloak will be able to produce anything remotely usable within acceptable time.
Considering that we also need to reinvent the whole ML implementation from scratch, which would likely be less mature, less feature rich, less reliable, and less performant (even without considering the Aircloak performance), I'm not convinced that this is the way to go.
Proxying off-the-shelf ML implementations in the Cloak
In this approach we would somehow allow analysts to produce the prediction model directly in the cloak. Hence, instead of providing an external tool, the analyst would issue a query, such as
CREATE PREDICTION MODEL FROM ....Internally, the cloak would fetch the raw data, and then feed it to some off-the-shelf solution. We would likely need to support different algorithms, and various options. The outcome of the action is a prediction model which can be fed with new inputs to obtain predictions.
The question here is can we safely return the prediction model to the analyst, i.e. does prediction model keep privacy guarantees. My current impression is that it doesn't, at least not for random forests, where each particular input combination is covered in some tree.
One way of dealing with this is to keep the model stored inside the cloak. This would mean that the analysts can only obtain predictions by sending new inputs to cloak. I'm not sure how useful would this be in practice though. More likely, analysts are accustomed to using their own tools, which are likely feature rich beyond what we can achieve with a reasonable effort.
Another option is to try and sanitize the obtained prediction model before sending it back. The purpose of the sanitization is to remove all the branches which can compromise an individual (or a small group). I'm not completely sure whether this resolves all the privacy issues though.
Maximizing the useful data returned from regular queries
As mentioned earlier, the root cause of or ML-related problem is the fact that with
select *usually all (or most) of the data is anonymized away, so the result becomes useless for further training.Instead of trying to provide an out-of-the-box ML solution, we could focus on returning as much of the useful data as possible. We already have a very basic hardcoded support for that, by premasking user_id fields, so that
select * from tablemight still end up returning some meaningful data, even though every selected row is unique.In the extension of this idea, the cloak would somehow figure out which fields should be premasked to increase the amount of the non-masked data returned. The benefit is then that analysts can issue their
selectfrom an arbitrary table (or join), get their data and use the result with any desired tool for any purpose, such as supervised or unsupervised ML. In other words, by improving the usefulness of the returned set, we can solve many problems.On the flipside, this will probably also require a sophisticated implementation (which might even end up using some of the ML techniques itself), it would also have its own performance penalty, and ultimately might not give optimal results.
That said, I feel that this approach has by far the best value-to-effort ratio compared to the previous two.