Fix for unexpected batch processor crash

[mihailefter]

Problem

It looks like the batch processor crashes in the __alterBatchEntries function. Considering an input file with the following contents:

• NM_024690:c.40638_40638delinsCTGGA
  • NM_024690:c.3609_3788AAGTAATATTCCAACAAGTGGTGCCATAGGAAAAAGCACCCTGGTTCCCTTGGACACTCCATCTCCAGCCACATCATTGGAGGCATCAGAAGGGGGACTTCCACCCTCAGCACCTACCCTGAATCAACAAACACACCCAGCATCCACCTCGGAGCACACGCTAGTTCAGAAAGTCCG

During the processing of the first job (which proceeds without crashing) Mutalyzer fetches the most recent version for NM_024690, which is NM_024690.2. Next it tries to update any other entries in the batch_queue_items database table which utilize only NM_024690 to the most recent version. This is done in order to speed up the batch process when those jobs are reached. In this case it tries to update the second job. The information stored in the item column of the batch_queue_items table for the second job has 200 characters, which is to be replaced by a larger one, of 202 characters. Since this is greater than the maximum allowed, the query results in an error:

(psycopg2.DataError) value too long for type character varying(200)

It seems that an input line is automatically truncated to 200 characters when added to the database, so no error appears there, but during the replace operation the truncation is no longer performed.

Possible solutions

1. Change the item column type in batch_queue_items table to a variable unlimited length type. This is supported by PostgreSQL as type text but didn't check for other SQL database management systems.
2. Don't process (skip) entries which are longer than 200 chars and do not perform the replacement query on them.

[mihailefter]

We went for the second solution in order not to alter the database. We implemented the following:

• Before adding a line to the database we check if its length is greater than 190. If so we flag it with 'S2'. In this manner the corresponding line will not pe processed.
• We changed the query in the __alterBatchEntries function not to alter entries which have the 'S2' flag.
• In the __processFlags function we check for the 'S2' flag and output a message to mention that the line was not processed since it had a length greater than 190.

[mihailefter]

We also discovered that "__alterBatchEntries" changes the input sequence when one accession is a substring of another accession which appears later in the file, since the first part of the latest one will be replaced.

Input file sequence:

NM_001315:c.723_723delinsAC
NM_001315507:c.1826_1831G

During the processing of NM_001315:c.723_723delinsAC the NM_001315507:c.1826_1831G is replaced to NM_001315.2507:c.1826_1831G.