Synthetic data workshop

[martintoreilly]

Our contributions:

Presentation on QUIPP

• 15 mins presentation
• 5 minute follow on discussion
• Use first part of CW20 presentation?

Round table discussion at the end

• 20-40 minutes
• Evaluation metrics + future directions

Please write down ideas here about the round table discussion and the presentation. Also you can edit the slides directly here: https://thealanturininstitute.sharepoint.com/:p:/s/QUIPP/EYuOMc69zxZPjD2dkNqou_YB_Yh0PTGg9M4jF-9mIXMddQ?e=FqUQf7

[gmingas]

Round table discussion questions

Prioritise the following three in break-out rooms:

Applications:

• Can you give examples where you have used synthetic data, or tried to but it was not successful?
  • Follow up: What was the reason you considered it in the first place and what did you use it for?
  • Follow up: How did synthesis fit around processes of data access and disclosure?
  • Follow up: What were the main challenges?

Privacy:

• How do data publishing organisations/providers assess the adequecy of privacy-preserving methods?
  • Follow up: To what extent are differential privacy, plausible deniability, k-anonymity and motivated intruder metrics accepted guarantees when publishing datasets? Are they understood?
  • How do we know what is good enough?
  • What are organisations protecting against (threat model)?
• Which privacy metrics are you interested in or using?
  • What are the challenges, pros, cons (scalability, utility, trustworthiness)?
  • Are you using data-centric metrics or inherent differential privacy (during synthesis)? Definition: Data-driven is a metric that is decoupled from the synthetic algorithm, i.e. it can be evaluated using only the original and synthetic datasets. This is in contrast to method-dependent privacy which happens when a synthetic algorithm is private by design.

Other questions (for break-out or common room):

Synthesis algorithms (generative models):

• Which synthesis algorithms have you used in realistic settings in industry, public sector and academia?
  • Specifically for tabular data, time series data, spatial data, graph data.
  • Types of algorithms apart from GANs, VAEs, multiple imputation/model-based and microsimulation?
  • What are the biggest pros/cons for each synthetic/generative model type?

Utility:

• Are synthetic data users interested in overall utility/similarity of the dataset or utility for a specific task (e.g. a specific model they want to train)?
  • How does the assessment of utility depend on the tasks performed with the synthetic data?
• What utility metrics have you found useful in your area?
  • Follow up: Specifically for particular types of data, e.g. graph, time series, geolocation, etc

Synthetic data collaboration:

• Would you be interested in attending a regular Turing event on synthetic data?
• How do you envisage a hackathon for synthetic data, given limitations in data sharing?

[gmingas]

Discussion format Also, regarding the format, we know that 37 people have accepted the invite, therefore it is likely that there will be at least 10-20 in the discussion at the end (20-25 according to Derek's estimate).

After discussing with Oliver, these are a few things we could do:

• Start with a few questions to everyone where we can visualise the answers live using an app. These could help warm up the discussion and also we can choose them so that we get useful info for QUIPP specifically (5 minutes).
• Separate people into breakout rooms (with 7-8 people per room) with one QUIPP moderator in each. Ask the same questions to all groups and keep notes in a common hackmd (similar to recent EDI workshop) (20 minutes).
• Bring everyone together (15 minutes) to:
  • Present main findings from each group.
  • Finish any conversations which are important for most people.
  • Allow time for free questions/discussion from anyone and from Slido.
  • Discuss the possibility of a new Turing interest group or a second/regular meetup.

[gmingas]

Live poll questions:

• Why are you interested in synthetic data? --> multiple choice (researcher developing generators/metrics, want to release data, want to perform analysis on sensitive data, other)
• What software/tools are you using for releasing sensitive data (including "writing my own code")?
• Which types of data generating models are you interested in or using? --> multiple choice (...)
• What privacy metric are you interested in or using? --> word-cloud
• What utility metrics are useful for the type of data you work with? --> word-cloud
• Would you be interested in a regular synthetic data meetup or interest group? --> yes/no

[gmingas]

Links to documents generated during the workshop:

• Notes from the talks here
• Answers to poll questions here
• Round table questions and notes here
• The event was recorded