Open user31415 opened 11 years ago
One more change to the modified line # 890 above: FROM: output_file = csv.writer(open(options.output, 'wb'), dialect=csv.excel,quoting=1) TO: output_file = csv.writer(open(options.output, 'wb'), delimiter = delimiter, quoting=csv.QUOTE_MINIMAL) # Defined delimiter and remove most quotes for SQLite.
Greetings,
I failed to include this awhile back. Could we get these updates included in the current version? They're definitely valuable.
Thanks.
-David
after Line # 267: SIAttributeSizeNT = 48
add delimiter = "|" # CSV definable delimiter because the data contains commas
modified line # 636 FROM: if ( p == 0 ) or ( p == 5 ): TO: if ( p == 0 ) or ( p >4 ): # probably unnecessary but I was looking for a spurious generated "1" when 4 filenames
modified Line # 663-677 Make headers DB friendly: Replace spaces with underscores, replace objectionable symbols (# & /), prefix FN1...4 to make headers unique.
after line # 716: # If this goes above four FN attributes ... add: # Limit to 4 add: if MFTR['fncnt'] > 4: MFTR['fncnt'] = 4
after line # 723: # Pad out the remaining FN columns add: tmpBuffer = [] # Initialize to empty. *\ This was the cause of the spurious "1"
modified line # 890: FROM: output_file = csv.writer(open(options.output, 'wb'), dialect=csv.excel,quoting=1) TO: output_file = csv.writer(open(options.output, 'wb'), delimiter = delimiter, quoting=1) # Defined delimiter
The above allowed me to successfully import a little under 1/2 million records into SQLite.