Closed galamit86 closed 3 years ago
@galamit86 You may want to consider including a more generic clean_names function where we could add more logic as we go along.
See https://gist.github.com/dkapitan/89ff20eeed38e6d9757fef9e09e23c3d
I have written than in the past to get rid of spaces. We could at a line to replace periods
@dkapitan Added generic function based on yours, with 2 differences:
s
instead of s.lower()
- I tend to stick to the original names as far as possible. Additionally, since this is is not applied on all tables, but only on DataProperties
, I think it's better not to lowercase only some tables.@galamit86 Looks good to me!
This PR:
main
), where theDataProperties
table column names are checked for any "." and replaced with "_". Ideally this would be done for all tables, but since there is no simple way to do this without loading the whole table to memory, this focuses on where this problem has actually occurred.Adds a function
fix_data_properties()
that:DataProperties
table from the latest GCS folderthereby fixing the issue without having to re-upload the whole dataset.