qgis / QGIS

QGIS is a free, open source, cross platform (lin/win/mac) geographical information system (GIS)
https://qgis.org
GNU General Public License v2.0
10.65k stars 3.01k forks source link

"import delimited text layer" does not work with files created on excel/osx #17190

Closed qgib closed 5 years ago

qgib commented 11 years ago

Author Name: Arthur Vignacq (Arthur Vignacq) Original Redmine Issue: 8421 Affected QGIS version: master Redmine category:data_provider/delimitedtext


Hello,

I just tried to import a CSV file in QGIS 1.9 master. But even if I select ; as the separator factor, QGIS just think everything is on the same row.

The import of the same file work very well in 1.8

Cheers


qgib commented 11 years ago

Author Name: Giovanni Manghi (@gioman)


did you choose custom delimiter -> semicolon?


qgib commented 11 years ago

Author Name: Arthur Vignacq (Arthur Vignacq)


Giovanni Manghi wrote:

did you choose custom delimiter -> semicolon?

I don't have this option. In custom delimiter I can choose between; "," "tab" "space" "colon" and ";" Neither work. All the content is place in 1st row. the exact same file with same procedure work fine in 1.8

qgib commented 11 years ago

Author Name: Giovanni Manghi (@gioman)


Arthur Vignacq wrote:

Giovanni Manghi wrote:

did you choose custom delimiter -> semicolon?

I don't have this option. In custom delimiter I can choose between; "," "tab" "space" "colon" and ";" Neither work. All the content is place in 1st row. the exact same file with same procedure work fine in 1.8

see attached image.

PS Are you sure you are running the latest master release?



qgib commented 11 years ago

Author Name: Arthur Vignacq (Arthur Vignacq)


Yes, I think it's the last release as I downloaded it yesterday (Mac osX).

My visual attached.



qgib commented 11 years ago

Author Name: Giovanni Manghi (@gioman)


Arthur Vignacq wrote:

Yes, I think it's the last release as I downloaded it yesterday (Mac osX).

My visual attached.

please attach a sample of you data

qgib commented 11 years ago

Author Name: Arthur Vignacq (Arthur Vignacq)


please see attached file



qgib commented 11 years ago

Author Name: Regis Haubourg (@haubourg)


Your field nom as sometime text delimiters, sometimes not. Could you try to export csv with another tool and see if it changes something? I had many troubles here to export correctly csv for all possible cases (encoding, decimal delimiter, text delimiter.. )

qgib commented 11 years ago

Author Name: Giovanni Manghi (@gioman)


regis Haubourg wrote:

Your field nom as sometime text delimiters, sometimes not. Could you try to export csv with another tool and see if it changes something? I had many troubles here to export correctly csv for all possible cases (encoding, decimal delimiter, text delimiter.. )

saving the file to libreoffice calc and then resaving it as csv make the resulting file work. My guess is that there is something wrong on how the file was originally created, I remember that the OsX for example had problems of this kind.

qgib commented 11 years ago

Author Name: Giovanni Manghi (@gioman)


The issue seems to be this one

http://developmentality.wordpress.com/2010/12/06/excel-2008-for-macs-csv-bug/

giovanni@sibirica ~/Downloads $ file XY_Ports_VNF.csv XY_Ports_VNF.csv: ASCII text, with CR line terminators

when resaved with libreoffice

giovanni@sibirica ~/Downloads $ file test2.csv test2.csv: ASCII text

please check and report back.

qgib commented 11 years ago

Author Name: Arthur Vignacq (Arthur Vignacq)


The weird thing, is that QGIS 1.8, perfectly read this same file with the possible mistake office mac made. I'll try it with the 2011 version of Microsoft office in few minutes

qgib commented 11 years ago

Author Name: Giovanni Manghi (@gioman)


Arthur Vignacq wrote:

The weird thing, is that QGIS 1.8, perfectly read this same file with the possible mistake office mac made.

the tool has been completely rewritten, so no surprise that now works this way.

qgib commented 11 years ago

Author Name: Arthur Vignacq (Arthur Vignacq)


Just used Office 2011. Same problem when saved as .. "csv " Re-tried with a save as " windows _ csv". This time QGIS read it fine. Although 3 element coming with 2x " at beginning and ending of name ? I've seen no reason for that.

qgib commented 11 years ago

Author Name: Regis Haubourg (@haubourg)


Those with quotes are cells that contain carriage returns it seems. Carriage returns an csv are really a mess. After large testing, we now use here xls 2003 as a pivot format for attribute data. (but it needs good cleaning (no formula, no encoded field names). In my theorical roadmap, I would like to fund refactoring of vector layer dialog to give impor options and preview, just like csv plugin, but for all data types. that would help. Anyway, csv and carriage return will always be a mess since there is no CSV norm..

qgib commented 11 years ago

Author Name: Giovanni Manghi (@gioman)


regis Haubourg wrote:

Anyway, csv and carriage return will always be a mess since there is no CSV norm..

Then I would propose to close this ticket, or eventually make it low priority and change its title (please propose something).

qgib commented 11 years ago

Author Name: Giovanni Manghi (@gioman)


qgib commented 10 years ago

Author Name: Alexander Bruy (@alexbruy)


qgib commented 10 years ago

Author Name: John Tull (John Tull)


I don't know if this could present a potential solution, but you could use dos2unix (i.e., 'dos2unix -c mac foo.csv') to preprocess cdv files on OSX. For the uninitiated, trying to add csv layers with mac line encodings causes qgis to freeze requiring a force quit. Using dos2unix would add a dependency for the mac platform, but this seems preferable to crashing the program when using a feature that is pretty commonly needed.

To implement, there would need to be an OS check to force OSX to run this operation. I can only assume that this is doable. FWIW, dos2unix is available via homebrew.

http://waterlan.home.xs4all.nl/dos2unix.html

qgib commented 10 years ago

Author Name: mac joost (mac joost)


For me, the issue importing CSV files on a Mac is that the importer doesn't support the default mac "record separator" (the line ending).

On a mac, the default line eding is CR (carriage return).

Unix line endings (LF: line feed) and Windows line endings (CRLF) are supported.

Changing the line ending of your csv file to eg. CR will make QGIS import the file just fine (I used TextWrangler).

It would be nice though if the importer could just support CR line endings or have it configurable.

qgib commented 10 years ago

Author Name: Giovanni Manghi (@gioman)


qgib commented 9 years ago

Author Name: Dewey Dunnington (@paleolimbot)


I realize this issue is over 5 months old, but I'm about to use QGIS to teach a GIS short course to 40 or so people who will use it for various basic biology and geology map making and some basic analysis. I managed to convince our department to choose QGIS instead of Arc because it's become stable and for anything we use it for on a daily basis it does everything we need. This particular issue is a major, major stumbling block in getting people to adopt the program, because the easiest way to get our field data into QGIS is by making a csv. Anybody with a mac in our department (which is almost everybody) does this with Excel 2008 or 2011, which both have the problem of separating lines with carriage returns instead of newline characters. It seems as though this issue is an extraordinarily easy fix (separate lines by \ \ , then by \ , then by \ ), and I realize there are more exciting features and developers only have so much time. However, I would just like to reiterate how important this very simple issue is to the adoption of QGIS. Nobody in our department can use the beautiful new symbology editor if they can't get their data into the program.

qgib commented 9 years ago

Author Name: Benjamin Warr (Benjamin Warr)


I repeat the plea made by Dewey. The standard or 'default' encoding for csv imports appears to be UTF-8, please can someone ensure that csv imports can be made on macs's !

qgib commented 9 years ago

Author Name: Giovanni Manghi (@gioman)


Benjamin Warr wrote:

I repeat the plea made by Dewey. The standard or 'default' encoding for csv imports appears to be UTF-8, please can someone ensure that csv imports can be made on macs's !

If this fix is so vital for you (and in general for Mac users) then please consider supporting its fix.

qgib commented 9 years ago

Author Name: Benjamin Warr (Benjamin Warr)


Hi Giovanni, my apologies for a newbie question, but I am not sure how to "support the fix". Ben

qgib commented 9 years ago

Author Name: Giovanni Manghi (@gioman)


Benjamin Warr wrote:

Hi Giovanni, my apologies for a newbie question, but I am not sure how to "support the fix". Ben

Hire a developer, see for example

http://qgis.org/en/site/forusers/commercial_support.html

qgib commented 7 years ago

Author Name: Giovanni Manghi (@gioman)


qgib commented 5 years ago

Author Name: Giovanni Manghi (@gioman)


End of life notice: QGIS 2.18 LTR Source: http://blog.qgis.org/2019/03/09/end-of-life-notice-qgis-2-18-ltr/