FAA has gone to 28 day files

[ssokol]

Jessie,

Looks like the FAA has struck again. Please see:

https://nfdc.faa.gov/xwiki/bin/view/NFDC/56+Day+NASR+Subscription

PLEASE NOTE

- NASR subscriber files are now being released every 28 days, rather than every 56 days.

- The last 56 Day NASR subscriber files cover the period March 2, 2017 through April 27, 2017.

- The first 28 Day NASR subscriber files cover the period March 30, 2017 through April 27, 2017.
Each set of subscriber files will be posted 28 days prior to the effective date. The files will contain data that is updated for that 28 day cycle and has met the cutoff for entry.

- By FAA policy and order, many NASR resources, generally categorized as "Enroute," are still updated on a 56 day charting basis. The following files will be included in the 28 day subscriber set, but will only change every 56 days: ATS, AWY, PFR, ARB, CDR, MTR, SSD, PJA, and STARDP, and WXL.

When I tried to process one of these files I got the following:

xxxx@vb-ubuntu:~/processFaaData$ ./create_databases.sh local_data/nfdc.faa.gov/webContent/28DaySub/28DaySubscription_Effective_2017-09-14.zip
---------- Recursively unzipping local_data/nfdc.faa.gov/webContent/28DaySub/28DaySubscription_Effective_2017-09-14.zip
No Special Use Airspace information found in /home/ssokol/processFaaData/local_data/nfdc.faa.gov/webContent/28DaySub/28DaySubscription_Effective_2017-09-14//Additional_Data/AIXM/SAA-AIXM_5_Schema/SaaSubscriberFile/SaaSubscriberFile/Saa_Sub_File

Any suggestions?

[ssokol]

Looks like there's a duplicate SaaSubscriberFile in the create_database script.

[ssokol]

Ok, made it part-way through the extract / build with a ton of warnings. Ended on an error:

/home/ssokol/processFaaData/local_data/nfdc.faa.gov/webContent/28DaySub/28DaySubscription_Effective_2017-09-14//Additional_Data/AIXM/SAA-AIXM_5_Schema/SaaSubscriberFile/Saa_Sub_File/R-4501D FORT LEONARD WOOD, MO.xml
WARNING: Datasource creation options ignored since an existing datasource
         being updated.
WARNING: Layer creation options ignored since an existing layer is
         being appended to.
WARNING: Layer creation options ignored since an existing layer is
         being appended to.
WARNING: Layer creation options ignored since an existing layer is
         being appended to.
WARNING: Layer creation options ignored since an existing layer is
         being appended to.
WARNING: Layer creation options ignored since an existing layer is
         being appended to.
WARNING: Layer creation options ignored since an existing layer is
         being appended to.
WARNING: Layer creation options ignored since an existing layer is
         being appended to.
-wrapdateline option only works when reprojecting to a geographic SRS
/home/ssokol/processFaaData/local_data/nfdc.faa.gov/webContent/28DaySub/28DaySubscription_Effective_2017-09-14//Additional_Data/AIXM/SAA-AIXM_5_Schema/SaaSubscriberFile/Saa_Sub_File/W-140E JACKSONVILLE, FL.xml
WARNING: Datasource creation options ignored since an existing datasource
         being updated.
WARNING: Layer creation options ignored since an existing layer is
         being appended to.
WARNING: Layer creation options ignored since an existing layer is
         being appended to.
WARNING: Layer creation options ignored since an existing layer is
         being appended to.
WARNING: Layer creation options ignored since an existing layer is
         being appended to.
WARNING: Layer creation options ignored since an existing layer is
         being appended to.
INFO: Open of `./special_use_airspace_spatialite.sqlite'
      using driver `SQLite' successful.
/home/ssokol/processFaaData/local_data/nfdc.faa.gov/webContent/28DaySub/28DaySubscription_Effective_2017-09-14//Additional_Data/Shape_Files/ClassArispace.shp
ERROR 1: sqlite3_step() failed:
  ClassArispace.GEOMETRY violates Geometry constraint [geom-type or SRID not allowed] (19)
ERROR 1: COMMIT transaction failed: cannot commit - no transaction is active
INFO: Open of `./controlled_airspace_spatialite.sqlite'
      using driver `SQLite' successful.
ssokol@sms-vb-ubuntu:~/processFaaData$

[jlmcgraw]

Hmm, I had previously updated scripts to get and work with the 28 day data and it had worked for me but I'll take a look tonight

On Tue, Aug 29, 2017 at 6:41 PM Steven Sokol notifications@github.com wrote:

Looks like there's a duplicate SaaSubscriberFile in the create_database script.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jlmcgraw/processFaaData/issues/2#issuecomment-325825191, or mute the thread https://github.com/notifications/unsubscribe-auth/ADSE3kE2u2SoKqoxMlvvKeuLM7Exc0xeks5sdJOLgaJpZM4PGozH .

[jlmcgraw]

I get a different error at the end

//Additional_Data/Shape_Files/ClassArispace.shp ERROR 1: Cannot insert feature with geometry of type MULTIPOLYGONZ in column GEOMETRY. Type POLYGONZ expected INFO: Open of ./controlled_airspace_spatialite.sqlite' using driverSQLite' successful.

But, I can still open the resulting spatialite database in QGIS and see correct looking controlled airspace.

Not sure what's going on

On 08/29/2017 06:50 PM, Steven Sokol wrote:

Ok, made it part-way through the extract / build with a ton of warnings. Ended on an error:

|/home/ssokol/processFaaData/local_data/nfdc.faa.gov/webContent/28DaySub/28DaySubscription_Effective_2017-09-14//Additional_Data/AIXM/SAA-AIXM_5_Schema/SaaSubscriberFile/Saa_Sub_File/R-4501D FORT LEONARD WOOD, MO.xml WARNING: Datasource creation options ignored since an existing datasource being updated. WARNING: Layer creation options ignored since an existing layer is being appended to. WARNING: Layer creation options ignored since an existing layer is being appended to. WARNING: Layer creation options ignored since an existing layer is being appended to. WARNING: Layer creation options ignored since an existing layer is being appended to. WARNING: Layer creation options ignored since an existing layer is being appended to. WARNING: Layer creation options ignored since an existing layer is being appended to. WARNING: Layer creation options ignored since an existing layer is being appended to. -wrapdateline option only works when reprojecting to a geographic SRS /home/ssokol/processFaaData/local_data/nfdc.faa.gov/webContent/28DaySub/28DaySubscription_Effective_2017-09-14//Additional_Data/AIXM/SAA-AIXM_5_Schema/SaaSubscriberFile/Saa_Sub_File/W-140E JACKSONVILLE, FL.xml WARNING: Datasource creation options ignored since an existing datasource being updated. WARNING: Layer creation options ignored since an existing layer is being appended to. WARNING: Layer creation options ignored since an existing layer is being appended to. WARNING: Layer creation options ignored since an existing layer is being appended to. WARNING: Layer creation options ignored since an existing layer is being appended to. WARNING: Layer creation options ignored since an existing layer is being appended to. INFO: Open of ./special_use_airspace_spatialite.sqlite' using driver SQLite' successful. /home/ssokol/processFaaData/local_data/nfdc.faa.gov/webContent/28DaySub/28DaySubscription_Effective_2017-09-14//Additional_Data/Shape_Files/ClassArispace.shp ERROR 1: sqlite3_step() failed: ClassArispace.GEOMETRY violates Geometry constraint [geom-type or SRID not allowed] (19) ERROR 1: COMMIT transaction failed: cannot commit - no transaction is active INFO: Open of ./controlled_airspace_spatialite.sqlite' using driver SQLite' successful. ssokol@sms-vb-ubuntu:~/processFaaData$ |

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jlmcgraw/processFaaData/issues/2#issuecomment-325826892, or mute the thread https://github.com/notifications/unsubscribe-auth/ADSE3tWzlbXRQSccip9wgBdtaBMLwgtPks5sdJXEgaJpZM4PGozH.

[ssokol]

Anything I can do to help debug the issue?

[ssokol]

Looks like the import scripts work ok until they try to digest the controlled airspace shape files, at which point a constraint is violated and everything grinds to a halt.

I've tried this on 28DaySubscription_Effective_2017-08-17.zip and on 28DaySubscription_Effective_2017-09-14.zip with the same results. Unfortunately, my knowledge of both perl and Spatialite is very limited, so don't understand what exactly is failing. Here's the error message again from the attempt to process the August file:

ERROR 1: sqlite3_step() failed:
  ClassArispace.GEOMETRY violates Geometry constraint [geom-type or SRID not allowed] (19)
ERROR 1: COMMIT transaction failed: cannot commit - no transaction is active

It looks like others have hit this error before:

https://groups.google.com/forum/#!topic/spatialite-users/MxLKFxpLu08 https://stackoverflow.com/questions/15203403/spatialite-v4-0-0-recovergeometrycolumn-violates-geometry-constraint-geom-type

I also get a large number of warnings like this:

WARNING: Layer creation options ignored since an existing layer is
         being appended to.
-wrapdateline option only works when reprojecting to a geographic SRS
WARNING: Datasource creation options ignored since an existing datasource
         being updated.

I seem to recall these or similar warnings happening with previous versions, so they may not be part of the problem.

[ssokol]

I just re-ran a much older (Feb 2016) FAA file and it processed fine. At the end of the run the controlled_airspace_spatialite.sqlite file contained all the expected data.

I then checked out what had changed, and it seems that the FAA is now putting all of the controlled airspace data into a single shape file rather than a set of separate files. Here's what you see in the Shape_Files directory of the older archives:

$ ls
class_b.dbf  class_b.shp  class_c.dbf  class_c.shp  class_d.dbf  class_d.shp  class_e0.dbf  class_e0.shp  class_e5.dbf  class_e5.shp
class_b.prj  class_b.shx  class_c.prj  class_c.shx  class_d.prj  class_d.shx  class_e0.prj  class_e0.shx  class_e5.prj  class_e5.shx

And here's what's in the newer files:

$ ls
ClassArispace.cpg  ClassArispace.dbf  ClassArispace.prj  ClassArispace.sbn  ClassArispace.sbx  ClassArispace.shp  ClassArispace.shp.xml  ClassArispace.shx
$

Looks like the change to a single new file is not compatible with ogr2ogr. It immediately chokes on the unified ClassArispace.shp file. (Is that a type on the FAA's part, or is "Arispace" something?)

Looks like the update was made in the August drop per:

CHANGE TO CLASS AIRSPACE SHAPEFILES 

    Beginning this 56 day cycle, 17 August 2017, instead of individual 
    shapefiles for Class B, C, D, E0, and E5 airspace, there will be one class 
    airspace shapefile. Attributes Class and Local_type will identify the 
    class airspace for each record.

See: https://nfdc.faa.gov/webContent/28DaySub/2017-08-17/README.txt

[ssokol]

Ah, looks like you fixed that several days ago. Doh. Closing this.

Thanks, -S