search

thenineteen / Semiology-Visualisation-Tool

Data driven 3D brain visualisation of semiology. Semiology to anatomy translator based on over 4600 patients from 309 peer-reviewed articles.
MIT License
9 stars 6 forks source link

Store files in a human readable format #17

Open fepegar opened 4 years ago

fepegar commented 4 years ago

Pickled files are not human-readable, and differences cannot be tracked by a version control system like git.

Also, can the Excel files be converted to CSV?

thenineteen commented 4 years ago

Agreed - I'm going to use the pickled files now for a future test fixture so won't touch them or update them. It could be converted to csv, but would have to change the code to read_csv and check the integrity of the loaded DataFrame in case there are formatting changes

thenineteen commented 4 years ago

can we leave excel file until it is fully finalised before converting to csv please,

there may be subtle changes which I don't want to manually check for until all data is gathered

fepegar commented 4 years ago

This also applies to the pickle files.

What do you mean by "subtle changes"?

If you are going to update the Excel only from time to time, it'd be great if we could use CSV instead of Excel, for the sake of the repository, e.g. to be able to see the diffs in the commits.

fepegar commented 4 years ago

Pickle’s nine flaws

thenineteen commented 4 years ago

as far as I can make out we have two pickles remaining:

DataFrames_Exclusions_df_postictalPEThyper_concordance_ET

and

DataFrames_Exclusions_dfETconc

and both are only called in (pseudo-)pseudo-code under scripts in slicer_run_query_semiology_lateralisation

I think they can all be removed?

fepegar commented 4 years ago

Yes.

thenineteen commented 4 years ago

all removed

changing excel to csv will need all read_excel and the header arguments etc to change

thenineteen commented 3 years ago

see comments in pull #194 - not sure it is worth the effort