Migrate format functions from sum-stats-formatter to gwas-sumstat-tools

[jiyue1214]

sum-stats-formatter is the old formatter which can handle different input file to format into read-to-submission format to GWAS catalog. We want to integrate these functions from sum-stats-formatter to gwas-sumstat-tools. Modules and CLI should be available

[jiyue1214]

1. One change in the new formatter is the absence of split by left/right name. This is because petl does not support this split directly, and it was unclear how it would benefit users.
2. Almost done, looking for some files to test.

[ljwh2]

@eks-ebi and @earlEBI to look for complicated files to test with (multiple changes needed, including in same column)

[jiyue1214]

In the gwas-sumstat-tools, Running the command gwas-ssf format can achieve three functions: Generate a configured file

• [x] Recognise potential separators
• [x] suggest header mapping
• [x] recognise the special character in the header and suggest it as one separator for the column

Apply the configure file:

• [x] Identify the new file separator
• [x] Split the specific column by separator or regex pattern, provide new column name for new columns
• [x] rename the column name
• [x] Find and replace values in the column (not the header)
• [x] extract a regex pattern from the column values (not the header)

Batch submission

• [x] Select batch submission via the LSF or Slurm

I tested one input file from Lizzy and it works great. I am testing it on other 7 real data.

[jiyue1214]

Working on test the gwas-ssf format on 9 studies (for AMP). Need to discuss with Laura how many studies is suitable to test the script.

[jiyue1214]

I tested the formatter on data from 4 publications for AMP (10 studies) and 5 studies from 5 publications from Lizzy, The current formatter can format files, but may missing some functions:

1. Cannot handle complex delimiter, one of the common examples is multiple whitespace. It is caused by petl can only read 1-character separator. Solution: using skipinitialspace=True can fix multiplespace.
2. Covert invalidate value: some cells could be invalidated (like -nan) and need to convert to #NA to pass the validation

Some other functions may be interested to think to integrate:

1. Filling the chr pos if the file only has rsid (we need to know user dbSNP and some reference, I may not suggest it as one function to UI) - decide not to include this feature

[jiyue1214]

Migration is finished and I improved the pipeline based on testing on the real data. This ticket can be closed.