search

CartoDB / cartodb

Location Intelligence & Data Visualization tool
http://carto.com
BSD 3-Clause "New" or "Revised" License
2.74k stars 654 forks source link

Add explanative errors from import error id's #1541

Closed iriberri closed 9 years ago

iriberri commented 9 years ago

Some basic ones, like for example: encoding, projections, columns with repeated names... we could retrieve the most frequent ones from the metrics and start with them.

Please @juanignaciosl let me know if there's anything that I could help you with. :smile:

cc/ @miguelarias

Repeated columns (not really, but strings are trimmed)

irb(main):001:0> DataImport['e97930bb-41e9-422a-a6d1-0f7c0d1e46c6'].log
=> #<CartoDB::Log @values={:id=>"6c12d4ab-08b9-462a-be33-fbe9906cf462", :type=>"import", :user_id=>"2597c0e4-68c4-4ace-a706-1ab0f47dea6e", :created_at=>2015-02-19 06:56:45 +0000, :updated_at=>2015-02-19 06:56:47 +0000, :entries=>"2015-02-19 06:56:45 UTC: Running on server que05.useb.cartodb.net with PID: 14925\n2015-02-19 06:56:45 UTC: new_importer()\n2015-02-19 06:56:45 UTC: Fetching datasource public_url metadata for item id https://s3.amazonaws.com/com.cartodb.imports.production/c7bb500d9ff501ec6225/nigeria_public_sanitation_data.csv?AWSAccessKeyId=AKIAJUI5EFFJIRZMEEMA&Expires=1424336205&Signature=Kyp3a11l%2Ftgvkf9aY2LSsgMnk6I%3D\n2015-02-19 06:56:46 UTC: File will be downloaded from https://s3.amazonaws.com/com.cartodb.imports.production/c7bb500d9ff501ec6225/nigeria_public_sanitation_data.csv?AWSAccessKeyId=AKIAJUI5EFFJIRZMEEMA&Expires=1424336205&Signature=Kyp3a11l%2Ftgvkf9aY2LSsgMnk6I%3D\n2015-02-19 06:56:46 UTC: Before importer run\n2015-02-19 06:56:46 UTC: Starting import for /tmp/imports/20150219-14925-ztfjyw/nigeria_public_sanitation_data.csv\n2015-02-19 06:56:46 UTC: Unpacking /tmp/imports/20150219-14925-ztfjyw/nigeria_public_sanitation_data.csv\n2015-02-19 06:56:46 UTC: Filename: /tmp/imports/20150219-14925-eh1t8j/nigeria_public_sanitation_data.csv Size (bytes): 6268596\n2015-02-19 06:56:46 UTC: Importing data from /tmp/imports/20150219-14925-eh1t8j/nigeria_public_sanitation_data.csv\n2015-02-19 06:56:46 UTC: File-based import load\n2015-02-19 06:56:47 UTC: Detected encoding ISO-8859-1\n2015-02-19 06:56:47 UTC: Using database connection with {:adapter=>\"postgres\", :encoding=>\"unicode\", :host=>\"10.0.128.16\", :database=>\"cartodb_user_2597c0e4-68c4-4ace-a706-1ab0f47dea6e_db\", :username=>\"postgres\", :port=>6432, :connect_timeout=>2.5, :user=>\"cartodb_user_2597c0e4-68c4-4ace-a706-1ab0f47dea6e\"}\n2015-02-19 06:56:47 UTC: ogr2ogr call:      OSM_USE_CUSTOM_INDEXING=NO PG_USE_COPY=YES PGCLIENTENCODING=ISO-8859-1  /usr/bin/ogr2ogr2 -f PostgreSQL -oo AUTODETECT_TYPE=YES -oo QUOTED_FIELDS_AS_STRING=YES PG:\"host=10.0.128.16 port=6432 user=cartodb_user_2597c0e4-68c4-4ace-a706-1ab0f47dea6e dbname=cartodb_user_2597c0e4-68c4-4ace-a706-1ab0f47dea6e_db password=1bdaf2451e9b0408a88b3598cc15d67f055de853cartodb_user_2597c0e4-68c4-4ace-a706-1ab0f47dea6e\"  -lco DIM=2 -lco PRECISION=NO /tmp/imports/20150219-14925-eh1t8j/nigeria_public_sanitation_data.csv  -nln cdb_importer.importer_77f97828b80411e481820e49835281d6 -nlt PROMOTE_TO_MULTI\n2015-02-19 06:56:47 UTC: ogr2ogr output:    Warning 1: Field 'Are there urinals in place?' already exists. Renaming it as 'are there urinals in place?2'\nWarning 1: Field 'Estimate the number of Male using the facility  daily' already exists. Renaming it as 'estimate the number of male using the facility  daily2'\nWarning 1: Field 'Estimate the number of Female using the facility  daily' already exists. Renaming it as 'estimate the number of female using the facility  daily2'\nWarning 1: Field 'Is there provision for the physically challenged persons?' already exists. Renaming it as 'is there provision for the physically challenged persons?2'\nWarning 1: Field 'If Yes, How much?' already exists. Renaming it as 'if yes, how much?2'\nWarning 1: Field 'Are there separate units for Female and Male?' already exists. Renaming it as 'are there separate units for female and male?2'\nWarning 1: Field 'Number of compartments for Male' already exists. Renaming it as 'number of compartments for male2'\nWarning 1: Field 'Number of compartments for Female' already exists. Renaming it as 'number of compartments for female2'\nWarning 1: Field 'Estimate the number of Male using the facility daily' already exists. Renaming it as 'estimate the number of male using the facility daily2'\nWarning 1: Field 'Estimate the number of Female using the facility daily' already exists. Renaming it as 'estimate the number of female using the facility daily2'\nWarning 1: Field 'Is there Environmental Health club in place?' already exists. Renaming it as 'is there environmental health club in place?2'\nWarning 1: Field 'Is the club functional?' already exists. Renaming it as 'is the club functional?2'\nWarning 1: Field 'Who maintains the facility?' already exists. Renaming it as 'who maintains the facility?2'\nWarning 1: Field 'Do you pay for the toilet waste disposal?' already exists. Renaming it as 'do you pay for the toilet waste disposal?2'\nERROR 1: ERROR:  column \"is there hand_washing facility available for users?\" specified more than once\n\nERROR 1: CREATE TABLE \"cdb_importer\".\"importer_77f97828b80411e481820e49835281d6\" ( ogc_fid SERIAL, PRIMARY KEY (ogc_fid), \"identifier\" VARCHAR, \"display name\" VARCHAR, \"instance\" INTEGER, \"submission date\" VARCHAR, \"submitter\" VARCHAR, \"duration\" time, \"name of enumerator\" VARCHAR, \"phone number of enumerator\" FLOAT8, \"name of respondent\" VARCHAR, \"phone number of the respondent\" FLOAT8, \"status / rank of the respondent\" VARCHAR, \"state\" VARCHAR, \"lga\" VARCHAR, \"town/city\" VARCHAR, \"community\" VARCHAR, \"latitude\" FLOAT8, \"longitude\" FLOAT8, \"elevation\" VARCHAR, \"geo code\" VARCHAR, \"type of institution\" VARCHAR, \"name of institution/facility\" VARCHAR, \"do you have sanitation facilities?\" BOOLEAN, \"select institution\" VARCHAR, \"type of toilet facility\" VARCHAR, \"photo of the facility\" VARCHAR, \"photo of the facility 2\" VARCHAR, \"photo of the facility3\" VARCHAR, \"photo of the facility 4\" VARCHAR, \"are there separate units for female and male\" BOOLEAN, \"number of compartments for male\" INTEGER, \"number of compartments for female\" INTEGER, \"is the facility currently in use?\" BOOLEAN, \"estimate the number of male using the facility  daily\" INTEGER, \"estimate the number of female using the facility  daily\" INTEGER, \"is there hand_washing facility available for users?\" BOOLEAN, \"is the hand_washing facility being used?\" BOOLEAN, \"are there urinals in place?\" BOOLEAN, \"is there provision for the physically challenged persons?\" BOOLEAN, \"what is the source of water for the sanitation facility:\" VARCHAR, \"is there committee in place to manage the facility?\" BOOLEAN, \"is the committee functional?\" BOOLEAN, \"do they pay for using the facility?\" BOOLEAN, \"are people employed to manage the facility?\" BOOLEAN, \"if yes, how many?\" INTEGER, \"is there waste_bin for the toilet?\" BOOLEAN, \"how frequent do you dispose the waste_bin?\" VARCHAR, \"how do you dispose the waste bin?\" VARCHAR, \"do you pay for the toilet waste disposal?\" BOOLEAN, \"if yes, how much?\" INTEGER, \"who manages the facility\" VARCHAR, \"are there separate units for female and male?\" BOOLEAN, \"no of compartments for male\" INTEGER, \"no of compartments for female\" INTEGER, \"estimate the number of male using the facility daily\" INTEGER, \"estimate the number of female using the facility daily\" INTEGER, \"are there separate facility for female and male teachers?\" BOOLEAN, \"are there urinals in place?2\" BOOLEAN, \"if yes, are there separate urinals for male and females?\" BOOLEAN, \"is there hand_washing facility available for users?\" BOOLEAN, \"is the hand_washing facility being used?\" BOOLEAN, \"estimate the number of male using the facility  daily2\" INTEGER, \"estimate the number of female using the facility  daily2\" INTEGER, \"is there provision for the physically challenged persons?2\" BOOLEAN, \"what is the source of water\" VARCHAR, \"is there environmental health club in place?\" BOOLEAN, \"is the club functional?\" BOOLEAN, \"who maintains the facility?\" VARCHAR, \"is there waste_bin for the toilet?\" BOOLEAN, \"how is the waste_bin disposed?\" VARCHAR, \"how frequent do you dispose the waste_bin\" VARCHAR, \"do you pay for the waste disposal?\" BOOLEAN, \"if yes, how much?2\" INTEGER, \"who manage the facility\" VARCHAR, \"are there separate units for female and male?2\" BOOLEAN, \"number of compartments for male2\" INTEGER, \"number of compartments for female2\" INTEGER, \"if no, total number of compartments\" INTEGER, \"is the facility currently in use\" BOOLEAN, \"estimate the number of male using the facility daily2\" INTEGER, \"estimate the number of female using the facility daily2\" INTEGER, \"is there hand_washing facility available for users?\" BOOLEAN, \"is the hand_washing facility being used?\" BOOLEAN, \"is there provision for the physically challenged persons\" BOOLEAN, \"what is the source of water?\" VARCHAR, \"is there environmental health club in place?2\" BOOLEAN, \"is the club functional?2\" BOOLEAN, \"who maintains the facility?2\" VARCHAR, \"how is the waste_bin disposed?\" VARCHAR, \"do you pay for the toilet waste disposal?2\" BOOLEAN, \"if yes, how much\" VARCHAR, \"how frequent do you dispose the waste bin\" VARCHAR, \"field_92\" VARCHAR, \"wkb_geometry\" geometry(GEOMETRY) )\nERROR:  column \"is there hand_washing facility available for users?\" specified more than once\n\nERROR 1: Unable to write feature 1 from layer nigeria_public_sanitation_data.\n\nERROR 1: Terminating translation prematurely after failed\ntranslation of layer nigeria_public_sanitation_data (use -skipfailures to skip errors)\n\n\n2015-02-19 06:56:47 UTC: ogr2ogr exit code: 256\n2015-02-19 06:56:47 UTC: Errored importing data from /tmp/imports/20150219-14925-eh1t8j/nigeria_public_sanitation_data.csv:\n2015-02-19 06:56:47 UTC: CartoDB::Importer2::LoadError: #<CartoDB::Log:0x00000007280fd0>\n2015-02-19 06:56:47 UTC: ---------------------------------------------------

Table without a header column (a float number is being detected as header)

-12-18 22:44:51 UTC: Proceeding to register
2014-12-18 22:44:51 UTC: Before renaming from importer_5eb5d24a870711e49bdc0e8dde98a187 to cities1000
2014-12-18 22:44:51 UTC: Before moving schema 'cities1000' from cdb_importer to public
2014-12-18 22:44:51 UTC: Before persisting metadata 'cities1000' data_import_id: edf64de7-0632-4cd6-99b7-ce7a1c102f74
2014-12-18 22:44:51 UTC: Exception: PG::Error: ERROR:  function pg_catalog.btrim(double precision) does not exist
LINE 4:                 'POINT(' || trim(\"longitude\") || ' ' || trim...
                                   ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.
2014-12-04 12:51:39 UTC: File-based import load
2014-12-04 12:51:39 UTC: Detected encoding ISO-8859-1
2014-12-04 12:51:39 UTC: Using database connection with {:adapter=>"postgres", :encoding=>"unicode", :host=>"10.0.128.17", :database=>"cartodb_user_6b304b59-2c86-4d7b-a8a2-dc98a5a35d48_db", :username=>"postgres", :port=>6432, :connect_timeout=>2.5, :user=>"cartodb_user_cf8c5cce-d573-450b-8c43-6c7eaf1dd80e"}
2014-12-04 12:51:39 UTC: ogr2ogr call:      OSM_USE_CUSTOM_INDEXING=NO PG_USE_COPY=YES PGCLIENTENCODING=ISO-8859-1  /usr/bin/ogr2ogr2 -f PostgreSQL -oo AUTODETECT_TYPE=YES -oo QUOTED_FIELDS_AS_STRING=NO PG:"host=10.0.128.17 port=6432 user=cartodb_user_cf8c5cce-d573-450b-8c43-6c7eaf1dd80e dbname=cartodb_user_6b304b59-2c86-4d7b-a8a2-dc98a5a35d48_db password=c58f89679fd483c91110c817f01374588bff5195cartodb_user_cf8c5cce-d573-450b-8c43-6c7eaf1dd80e"  -lco DIM=2 -lco PRECISION=NO /tmp/imports/20141204-10913-10zw7kc/state_bounds.csv  -nln cdb_importer.importer_4a2150387bb411e4ab900e5e07bb5d8a -nlt PROMOTE_TO_MULTI
2014-12-04 12:51:39 UTC: ogr2ogr output:    Segmentation fault (core dumped)

2014-12-04 12:51:39 UTC: ogr2ogr exit code: 35584
2014-12-04 12:51:39 UTC: Errored importing data from /tmp/imports/20141204-10913-10zw7kc/state_bounds.csv:
2014-12-04 12:51:39 UTC: CartoDB::Importer2::LoadError: #
2014-12-15 10:23:30 UTC: ogr2ogr call:      OSM_USE_CUSTOM_INDEXING=NO PG_USE_COPY=YES PGCLIENTENCODING=WIN1252 SHAPE_ENCODING=WIN1252 /usr/bin/ogr2ogr -f PostgreSQL  PG:"host=10.0.128.17 port=6432 user=cartodb_user_cf8c5cce-d573-450b-8c43-6c7eaf1dd80e dbname=cartodb_user_6b304b59-2c86-4d7b-a8a2-dc98a5a35d48_db password=c58f89679fd483c91110c817f01374588bff5195cartodb_user_cf8c5cce-d573-450b-8c43-6c7eaf1dd80e" -t_srs EPSG:4326  -lco DIM=2 -lco PRECISION=NO /tmp/imports/20141215-6806-1mur7rc/income_oakland/income_oakland.shp  -nln cdb_importer.importer_5f119aa0844411e4a43b0e43f3deba5a -nlt PROMOTE_TO_MULTI
2014-12-15 10:23:30 UTC: ogr2ogr output:    ERROR 6: No translation for Mercator_Auxiliary_Sphere to PROJ.4 format is known.
Failed to create coordinate transformation between the
following coordinate systems.  This may be because they
are not transformable, or because projection services
(PROJ.4 DLL/.so) could not be loaded.
Source:
PROJCS["WGS_1984_Web_Mercator_Auxiliary_Sphere",
    GEOGCS["GCS_WGS_1984",
        DATUM["WGS_1984",
            SPHEROID["WGS_84",6378137.0,298.257223563]],
        PRIMEM["Greenwich",0.0],
        UNIT["Degree",0.0174532925199433]],
    PROJECTION["Mercator_Auxiliary_Sphere"],
    PARAMETER["False_Easting",0.0],
    PARAMETER["False_Northing",0.0],
    PARAMETER["Central_Meridian",0.0],
    PARAMETER["Standard_Parallel_1",0.0],
    PARAMETER["Auxiliary_Sphere_Type",0.0],
    UNIT["Meter",1.0]]
Target:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0,
        AUTHORITY["EPSG","8901"]],
    UNIT["degree",0.0174532925199433,
        AUTHORITY["EPSG","9122"]],
    AUTHORITY["EPSG","4326"]]
ERROR 1: Terminating translation prematurely after failed
translation of layer income_oakland (use -skipfailures to skip errors)

2014-12-15 10:23:30 UTC: ogr2ogr exit code: 256
2014-12-15 10:23:30 UTC: Errored importing data from /tmp/imports/20141215-6806-1mur7rc/income_oakland/income_oakland.shp:
2014-12-15 10:23:30 UTC: CartoDB::Importer2::LoadError: #
2014-12-15 10:23:30 UTC: ----------------------------------------------------
2014-12-15 10:23:31 UTC: ----------------------------------------------------
2014-12-15 10:23:31 UTC: Proceeding to register
2014-12-15 10:23:31 UTC: After importer run
2014-12-15 10:23:31 UTC: After dispatch
2014-12-15 10:23:31 UTC: ERROR!
2014-12-17 10:45:16 UTC: Filename: /tmp/imports/20141217-21784-yzy2tc/parques/parques.shp Size (bytes): 2704
2014-12-17 10:45:16 UTC: Importing data from /tmp/imports/20141217-21784-yzy2tc/parques/parques.shp
2014-12-17 10:45:16 UTC: File-based import load
2014-12-17 10:45:16 UTC: Errored importing data from /tmp/imports/20141217-21784-yzy2tc/parques/parques.shp:
2014-12-17 10:45:16 UTC: CartoDB::Importer2::MissingProjectionError: CartoDB::Importer2::MissingProjectionError
2014-12-18 12:03:32 UTC: Georeferencing...
2014-12-18 12:03:32 UTC: Disabling autovacuum for "cdb_importer"."importer_bfaa8b0e86ad11e48ba50e6e1df11cbf"
2014-12-18 12:03:32 UTC: Creating the_geom from the_geom column
2014-12-18 12:03:32 UTC: replace empty strings by nulls?
2014-12-18 12:03:32 UTC: no string column found, nothing replaced
2014-12-18 12:03:32 UTC: geometrifying
2014-12-18 12:03:32 UTC: Error creating the_geom: PG::Error: ERROR:  invalid input syntax for type double precision: ""
LINE 4:           AND the_geom != ''
                                  ^

2014-12-18 12:03:32 UTC: Errored importing data from /tmp/imports/20141218-13594-dk1j86/lot_example.csv:
2014-12-18 12:03:32 UTC: Sequel::DatabaseError: PG::Error: ERROR:  invalid input syntax for type double precision: ""
LINE 4:           AND the_geom != ''
                                  ^

2014-12-18 12:03:32 UTC: ----------------------------------------------------

2014-12-18 12:03:32 UTC: ----------------------------------------------------
2014-12-18 12:03:32 UTC: Proceeding to register
2014-12-18 12:03:32 UTC: After importer run
2014-12-18 12:03:32 UTC: After dispatch
2014-12-18 12:03:32 UTC: ERROR!
2014-12-18 16:44:15 UTC: Detected encoding UTF-8
2014-12-18 16:44:15 UTC: Using database connection with {:adapter=>"postgres", :encoding=>"unicode", :host=>"10.0.128.17", :database=>"cartodb_user_6b304b59-2c86-4d7b-a8a2-dc98a5a35d48_db", :username=>"postgres", :port=>6432, :connect_timeout=>2.5, :user=>"cartodb_user_cf8c5cce-d573-450b-8c43-6c7eaf1dd80e"}
2014-12-18 16:44:15 UTC: ogr2ogr call:      OSM_USE_CUSTOM_INDEXING=NO PG_USE_COPY=YES PGCLIENTENCODING=UTF-8  /usr/bin/ogr2ogr -f PostgreSQL  PG:"host=10.0.128.17 port=6432 user=cartodb_user_cf8c5cce-d573-450b-8c43-6c7eaf1dd80e dbname=cartodb_user_6b304b59-2c86-4d7b-a8a2-dc98a5a35d48_db password=c58f89679fd483c91110c817f01374588bff5195cartodb_user_cf8c5cce-d573-450b-8c43-6c7eaf1dd80e" -t_srs EPSG:4326  -lco DIM=2 -lco PRECISION=NO /tmp/imports/20141218-28193-b7mpqk/test.json  -nln cdb_importer.importer_19fd43f486d511e4a66e0e5e07bb5d8a -nlt PROMOTE_TO_MULTI
2014-12-18 16:44:15 UTC: ogr2ogr output:    ERROR 4: Failed to read GeoJSON data
FAILURE:
Unable to open datasource `/tmp/imports/20141218-28193-b7mpqk/test.json' with the following drivers.
  -> ESRI Shapefile
  -> MapInfo File
  -> UK .NTF
  -> SDTS
  -> TIGER
  -> S57
  -> DGN
  -> VRT
  -> REC
  -> Memory
  -> BNA
  -> CSV
  -> NAS
  -> GML
  -> GPX
  -> LIBKML
  -> KML
  -> GeoJSON
  -> Interlis 1
  -> Interlis 2
  -> GMT
  -> GPKG
  -> SQLite
  -> DODS
  -> ODBC
  -> WAsP
  -> PGeo
  -> MSSQLSpatial
  -> PostgreSQL
  -> MySQL
  -> PCIDSK
  -> OpenFileGDB
  -> XPlane
  -> AVCBin
  -> AVCE00
  -> DXF
  -> Geoconcept
  -> GeoRSS
  -> GPSTrackMaker
  -> VFK
  -> PGDump
  -> OSM
  -> GPSBabel
  -> SUA
  -> OpenAir
  -> PDS
  -> WFS
  -> HTF
  -> AeronavFAA
  -> Geomedia
  -> EDIGEO
  -> GFT
  -> GME
  -> SVG
  -> CouchDB
  -> Idrisi
  -> ARCGEN
  -> SEGUKOOA
  -> SEGY
  -> XLS
  -> ODS
  -> XLSX
  -> ElasticSearch
  -> PDF
  -> Walk
  -> CartoDB
  -> SXF

2014-12-18 16:44:15 UTC: ogr2ogr exit code: 256
2014-12-18 16:44:15 UTC: Errored importing data from /tmp/imports/20141218-28193-b7mpqk/test.json:
2014-12-18 16:44:15 UTC: CartoDB::Importer2::UnsupportedFormatError: #
2014-12-18 16:44:15 UTC: ----------------------------------------------------
2014-12-30 09:14:53 UTC: File-based import load
2014-12-30 09:14:53 UTC: Detected encoding UTF-8
2014-12-30 09:14:53 UTC: Using database connection with {:adapter=>"postgres", :encoding=>"unicode", :host=>"10.0.128.17", :database=>"cartodb_user_6b304b59-2c86-4d7b-a8a2-dc98a5a35d48_db", :username=>"postgres", :port=>6432, :connect_timeout=>2.5, :user=>"cartodb_user_cf8c5cce-d573-450b-8c43-6c7eaf1dd80e"}
2014-12-30 09:14:53 UTC: ogr2ogr call:      OSM_USE_CUSTOM_INDEXING=NO PG_USE_COPY=YES PGCLIENTENCODING=UTF-8  /usr/bin/ogr2ogr -f PostgreSQL  PG:"host=10.0.128.17 port=6432 user=cartodb_user_cf8c5cce-d573-450b-8c43-6c7eaf1dd80e dbname=cartodb_user_6b304b59-2c86-4d7b-a8a2-dc98a5a35d48_db password=c58f89679fd483c91110c817f01374588bff5195cartodb_user_cf8c5cce-d573-450b-8c43-6c7eaf1dd80e" -t_srs EPSG:4326  -lco DIM=2 -lco PRECISION=NO /tmp/imports/20141230-5402-18g6pkq/ecf_current_maps.kml  -nln cdb_importer.importer_506166b8900411e488cf0e43f3deba5a -nlt PROMOTE_TO_MULTI
2014-12-30 09:14:53 UTC: ogr2ogr output:    ERROR 4: ERROR Parseing kml /tmp/imports/20141230-5402-18g6pkq/ecf_current_maps.kml :not well-formed (invalid token) on line 8945 at offset 0
FAILURE:
Unable to open datasource `/tmp/imports/20141230-5402-18g6pkq/ecf_current_maps.kml' with the following drivers.
  -> ESRI Shapefile
  -> MapInfo File
  -> UK .NTF
  -> SDTS
  -> TIGER
  -> S57
  -> DGN
  -> VRT
  -> REC
  -> Memory
  -> BNA
  -> CSV
  -> NAS
  -> GML
  -> GPX
  -> LIBKML
  -> KML
  -> GeoJSON
  -> Interlis 1
  -> Interlis 2
  -> GMT
  -> GPKG
  -> SQLite
  -> DODS
  -> ODBC
  -> WAsP
  -> PGeo
  -> MSSQLSpatial
  -> PostgreSQL
  -> MySQL
  -> PCIDSK
  -> OpenFileGDB
  -> XPlane
  -> AVCBin
  -> AVCE00
  -> DXF
  -> Geoconcept
  -> GeoRSS
  -> GPSTrackMaker
  -> VFK
  -> PGDump
  -> OSM
  -> GPSBabel
  -> SUA
  -> OpenAir
  -> PDS
  -> WFS
  -> HTF
  -> AeronavFAA
  -> Geomedia
  -> EDIGEO
  -> GFT
  -> GME
  -> SVG
  -> CouchDB
  -> Idrisi
  -> ARCGEN
  -> SEGUKOOA
  -> SEGY
  -> XLS
  -> ODS
  -> XLSX
  -> ElasticSearch
  -> PDF
  -> Walk
  -> CartoDB
  -> SXF

2014-12-30 09:14:53 UTC: ogr2ogr exit code: 256
2014-12-30 09:14:53 UTC: Errored importing data from /tmp/imports/20141230-5402-18g6pkq/ecf_current_maps.kml:
2014-12-30 09:14:53 UTC: CartoDB::Importer2::UnsupportedFormatError: #
2014-11-14 12:08:23 UTC: Processing first sheet
2014-11-14 12:08:23 UTC: Calculating columns (this can take long as will scan the full document)
2014-11-14 12:08:28 UTC: Calculating rows
2014-11-14 12:08:30 UTC: Newlines removed
2014-11-14 12:08:30 UTC: Orig file: /tmp/imports/20141114-30693-1t0zqlk/flowdata_charts_final2.xls
Temp destination: /tmp/imports/20141114-30693-1t0zqlk/flowdata_charts_final2.csv
2014-11-14 12:08:46 UTC: Detected encoding ISO-8859-1
2014-11-14 12:08:46 UTC: Using database connection with {:adapter=>"postgres", :encoding=>"unicode", :host=>"10.0.128.17", :database=>"cartodb_user_76996b45-ba54-450b-a960-02eee67479f6_db", :username=>"postgres", :port=>6432, :connect_timeout=>2.5, :user=>"cartodb_user_76996b45-ba54-450b-a960-02eee67479f6"}
2014-11-14 12:08:46 UTC: ogr2ogr call:      OSM_USE_CUSTOM_INDEXING=NO PG_USE_COPY=YES PGCLIENTENCODING=ISO-8859-1  /usr/bin/ogr2ogr2 -f PostgreSQL -oo AUTODETECT_TYPE=YES -oo QUOTED_FIELDS_AS_STRING=NO PG:"host=10.0.128.17 port=6432 user=cartodb_user_76996b45-ba54-450b-a960-02eee67479f6 dbname=cartodb_user_76996b45-ba54-450b-a960-02eee67479f6_db password=351abd03f451afe12b6bfd965e5a441e6a1f9387cartodb_user_76996b45-ba54-450b-a960-02eee67479f6"  -lco DIM=2 -lco PRECISION=NO /tmp/imports/20141114-30693-1t0zqlk/flowdata_charts_final2.csv  -nln cdb_importer.importer_ee33d5a66bf611e48b6a0e5e07bb5d8a -nlt PROMOTE_TO_MULTI
2014-11-14 12:08:46 UTC: ogr2ogr output:    
2014-11-14 12:08:46 UTC: ogr2ogr exit code: 0
2014-11-14 12:08:46 UTC: Georeferencing...
2014-11-14 12:08:46 UTC: Disabling autovacuum for "cdb_importer"."importer_ee33d5a66bf611e48b6a0e5e07bb5d8a"
2014-11-14 12:08:46 UTC: Identified lat as latitude column
2014-11-14 12:08:46 UTC: Identified long as longitude column
2014-11-14 12:08:46 UTC: Creating the_geom from latitude / longitude
2014-11-14 12:08:46 UTC: Creating the_geom column
2014-11-14 12:08:46 UTC: Populating the_geom from latitude / longitude
2014-11-14 12:08:47 UTC: Total processed: 5928
2014-11-14 12:08:47 UTC: Total processed: 5928
2014-11-14 12:08:47 UTC: FINISHED: Populating the_geom from latitude / longitude
2014-11-14 12:08:47 UTC: Enabling autovacuum for "cdb_importer"."importer_ee33d5a66bf611e48b6a0e5e07bb5d8a"
2014-11-14 12:08:47 UTC: Georeferenced
2014-11-14 12:08:47 UTC: Finished importing data from /tmp/imports/20141114-30693-1t0zqlk/flowdata_charts_final2.xls
2014-11-14 12:08:47 UTC: Proceeding to register
2014-11-14 12:08:47 UTC: Before renaming from importer_ee33d5a66bf611e48b6a0e5e07bb5d8a to flowdata_charts_final2
2014-11-14 12:08:47 UTC: Before moving schema 'flowdata_charts_final2' from cdb_importer to public
2014-11-14 12:08:47 UTC: Before persisting metadata 'flowdata_charts_final2' data_import_id: 2f9b5901-bd98-4b38-9cea-c380c997c2f6
2014-11-14 12:08:47 UTC: Exception: Connection timed out
2014-11-14 12:08:47 UTC: ERROR!
2014-12-30 17:00:18 UTC: Importing data from /tmp/imports/20141230-8916-1cn89s7/data_start_map.xlsx
2014-12-30 17:00:18 UTC: File-based import load
2014-12-30 17:00:18 UTC: Converting XLSX to CSV
2014-12-30 17:00:44 UTC: Removing newlines...
2014-12-30 17:00:44 UTC: Processing first sheet
2014-12-30 17:00:44 UTC: Calculating columns (this can take long as will scan the full document)
2014-12-30 17:05:29 UTC: Import timed out. Id:3a7510c1-666b-4c80-aea9-fb7b56b7d393 State:importing Created at:2014-12-30 17:00:13 +0000 Running imports:["ebf19d36-a674-4210-80af-f64a67187f06", "b2e48f3f-f10e-44ca-b7ae-b9fa0ee2b93c"]
2014-12-31 08:29:54 UTC: ERROR!
juanignaciosl commented 9 years ago

It'd be great if you can provide a test suite. Could you sort the ones you found for a couple of weeks from now on? It should not cost you much and it'd be really useful, because I can keep special care at the most common user errors.

iriberri commented 9 years ago

I'll try to get them. If I'm not able to get them all, I recommend you to search for examples in my own account: I usually test a lot of files so it's probable that I have most of the errors in my account. :-)

saleiva commented 9 years ago

Nothing to be designed here. The window supports better error reporting.

iriberri commented 9 years ago

Thanks! Just wanted to ask because don't know about its new design.

iriberri commented 9 years ago

I'm adding the cases I'm facing today to the first post.

santisaez commented 9 years ago

As talked via Slack, another suggestion regarding to the import process: warn the user from the editor if it's uploading an uncompressed file, advising that compressing the file will reduce the upload time.

iriberri commented 9 years ago

I have just seen an encoding error without explanation in the UI, wasn't that be added in this issue? I'm seeing "Encoding::InvalidByteSequenceError" but the system is not raising it. I have a file that I can share in private if needed.

juanignaciosl commented 9 years ago

@iriberri not all encoding errors are catched in the same place or in the same way. Open a new issue (with the trace if you have it) and send me the file.

iriberri commented 9 years ago

Done https://github.com/CartoDB/cartodb/issues/4355