search

subsurface / subsurface

This is the official upstream of the Subsurface divelog program
https://subsurface-divelog.org
GNU General Public License v2.0
2.62k stars 509 forks source link

Attempting Manual Import fails with no error #3115

Open johnsgill3 opened 3 years ago

johnsgill3 commented 3 years ago

Describe the issue:

Issue long description:

I have about 70 dives that I recorded previously in an excel spreadsheet prior to obtaining a dive computer. I am trying to import the data into subsurface but am having a really hard time doing it. Trying to import the raw csv data generates an error without anything other than "Failed to import". I wrote a little python program to convert some of the data in the CSV to better formats (like duration, and O2) and change the separator from comma (',') to a pipe ('|') since I had some fields (buddy/notes) that had comma's in them and it was throwing off the parser. However, I have been unable to get the import to work successfully and I get no error or any sort of log about why the import failed, or even which "dives"/"rows" are bad. It would be really helpful if when doing the import you could generate the importdive.log file that could be analyzed for any errors additionally any help on how to do successful imports would be appreciated.

Operating system:

OS X Big Sur (11.1)

MacBookPro

Subsurface version:

4.9.10

Have not tried

Official Release

Steps to reproduce:

The following is a sample of the data set.

1|2012-06-02|San Diego (La Jolla Shores)|||0|||||80|||21|B. Uddy1, B. Uddy2|NAUI Basic Cert Dive #1
2|2012-06-02|San Diego (La Jolla Shores)|||22|22||2900|1000|80|3000||21|B. Uddy1, B. Uddy2|NAUI Basic Cert Dive #2
3|2012-06-02|San Diego (La Jolla Shores)|||22|36||3000|1100|80|3000||21|B. Uddy1, B. Uddy2|NAUI Basic Cert Dive #3
4|2012-06-03|San Diego (La Jolla Shores)|||28|58||2900|0|80|3000||21|B. Uddy1, B. Uddy2|NAUI Basic Cert Dive #4
5|2012-06-03|San Diego (La Jolla Shores)|||25|35||3000|600|80|3000||21|B. Uddy1, B. Uddy2|NAUI Basic Cert Dive #5
6|2012-06-23|San Diego (La Jolla Shores)|||30|60||3000|500|80|3000||21|B. Uddy3|
7|2012-06-24|San Diego (La Jolla Shores)|||25|60||3000|500|80|3000||21|B. Uddy3|
8|2012-07-07|San Diego (La Jolla Cove)|07:45|08:25|40|50||3000|250|80|3000||21|B. Uddy3|
9|2012-07-07|San Diego (La Jolla Cove)|09:45|10:25|40|50||3000|500|80|3000||21|B. Uddy3|
10|2012-07-22|San Diego (La Jolla Shores)|07:30|08:15|45|40||3100|250|80|3000||21|B. Uddy3|
11|2012-07-22|San Diego (La Jolla Shores)|10:05|10:55|50|35||3000|500|80|3000||21|B. Uddy3|

Fields selected/set are as follows: Dive #|Date|Location|Time|Ignore|Duration|Start PSI|End PSI|Cylinder|Ignore|Ignore|O2|Buddy|Notes Separator: | Date: yyyy-mm-dd Time: Minutes Units: Imperial

Current behavior:

No error and no dives are imported

Expected behavior:

Some sort of useful error or successful import of dives

Additional information:

See sample data above

Mentions:

dirkhh commented 3 years ago

@mturkia - this looks like something for you to investigate? Of course it's winter holidays, and winter in Finland is, err, long... :-)

mturkia commented 3 years ago

Looks like pipe is not a working field separator. When using comma, you could use quoting to evade the problem with comma being part of the buddy/notes field. Excel should produce proper quoting, if you only enable this option.

Unfortunately we are using XSLT to import the CSV file and we either get valid output from it or we don't. To be able to give proper error messages, the whole import would need to be re-implemented with something with a parser in C or C++.

Winter is indeed long and dark over here. Perfect time for hibernation.

johnsgill3 commented 3 years ago

I was able to finally get it to work with a few things:

  1. I changed the python csv writer to use tab(\t) as the delimiter
  2. I found that I had some < and > characters in my notes and it was throwing off the XSLT parsing. Once I removed those things worked much smoother. But I had to do a lot of import 10 rows, import 5 rows to try and narrow down the offending row.
  3. If my first row in the data I was importing did not contain all possible columns - i.e. some of my dives I didn't put "Notes" and it being the last column. Then the table to assign fields did not give me an option to import this column.

I'll let you decide if there is still some options in here for improvements, but I think I have finally completed my import.

mturkia commented 3 years ago

I was pretty sure < and > were taken care of while pre-prosessing the input before XSLT parsing. But looking at the code, only ampersand is encoded in this phase. I will need to add these characters there as well. And I will have to take a look why the pipe character is not working as a delimiter.

When importing CSV, you could have a header line first, giving names for the columns. This way you would have all the fields there, and Subsurface would even try to assign the field configuration for you, if the names are somewhat standard. Will have to check our documentation to see if this is mentioned or not.