Closed crangelsmith closed 4 years ago
De-identified, multivariate correlated dataset (original): https://github.com/crangelsmith/synthetic-data-tutorial/blob/master/data/hospital_ae_data_deidentify.csv
Json:
attribute_to_datatype = {
'Time in A&E (mins)': 'Integer',
'Treatment': 'String',
'Gender': 'String',
'Index of Multiple Deprivation Decile': 'Integer',
'Hospital ID': 'String',
'Arrival Date': 'String',
'Arrival hour range': 'String',
'Age bracket': 'String'
}
attribute_is_categorical = {
'Hospital ID': True,
'Time in A&E (mins)': False,
'Treatment': True,
'Gender': True,
'Index of Multiple Deprivation Decile': False,
'Arrival Date': True,
'Arrival hour range': True,
'Age bracket': True
}
Kasra and I refactored the synthesis code pipeline under CTGAN (previously main functions were contained in utils.py
). We also added extra functionality, checks for correct user input and verbosity. The implementation is now in ctgan_main.py
. The proposed class structure for different synthesis methods is:
Base
class which contains:
read_data
function which reads the input data and the .json metadata (both could come from the generation script or from some other source) and returns them. It takes the filepaths for the data and metadata and the synth_name as arguments. Examples of the two input files can be found under CTGAN/tests/data
fit_synthesizer
and synthesize
. Base
. Each one implements a different synthesis method. Each class needs to contain:
read_data
function which can call read_data
from Base
and then can apply extra method-specific data pre-processing if needed. fit_synthesizer
: This calls _read_data
and then fits the model to the data using the chosen method (in our case CTGAN). Uses metadata from .json containing method-specific tuning parameters (example under CTGAN/tests/parameters
). It returns the fitted model and also stores it within the object.synthesize
: This takes the fitted model and uses it to synthesise the final dataset. This might also use some of the parameter metadata from .json but this needs to be discussed. Returns the final synthetic dataset and can optionally write the dataset to disk. At the moment this does not return output metadata yet.To run the code for the CTGAN case, go to the CTGAN sub-directory and run the following in bash (note the test code will eventually be removed from this file):
> python ctgan_main.py
You need to run pip install ctgan
if you do not have the library installed.
Photo of the whiteboard from yesterday's meeting where we discussed possible directory structures and .json file formats
The aim is to adapt at least one method to take each dataset in a standard format (csv + json), and emit synthetic data in a specified location
Requirements
synth-method
, make a subdirectory for each methodsynth-name
for the synthesis)synthetic-output/dataset-name/method-name
: