Exception: Output file is not shp or csv

rtsaboya commented 9 months ago

Hello! First of all, thanks for this incredible tool!

I am trying to generate a Skim Matrix but keep getting the following error:

Exception: Output file is not shp or csv

What I already tried:

I converted the layer to shapefile before running the tool.
I prepared the network and generated a small, simplified version.
I checked that there are no null values in the origin or destination fields.
I tested with Euclidean and Pedestrian distances.
I ran an integral analysis successfully using the same layer.
I checked that there is enough space on the disk.

If you have any clues as to what is causing the error, I would really appreciate to hear.

In case you want to check the actual layer, here is a link. The field for origins and destinations is "CD_GEOCO_1"

Thanks!

JamesParrott commented 9 months ago

I don't claim the results are meaningful, or that I've entered the correct command. But I didn't see this exception.

C:\...\sDNA_sample_layer_for_testing>py "c:\Program Files (x86)\sDNA\bin\sdnaintegral.py" -i temp4.shp -o out.shp outputskim;skimzone=CD_GEOCO_1
c:\Program Files (x86)\sDNA\sdnapy.py:30: SyntaxWarning: "is not" with 'str' literal. Did you mean "!="?
  if path is not "":
sDNA sdnaintegral config: outputskim;skimzone=CD_GEOCO_1
Loading DLL c:\Program Files (x86)\sDNA\\x64\\sdna_vs2008.dll
  Using default of angular link centres for angular analysis
  Using xytolerance=0, ztolerance=0
Input has 1815 rows
Creating geometry output out_skim
Creating geometry output out
Adding fields:
    ID         - ID
    CD_GEOCO_1 - CD_GEOCO_1
    LConn      - Line Connectivity
    LLen       - Line Length
    LFrac      - Link Fraction
    LAC        - Line Ang Curvature
    LSin       - Line Sinuosity
    LBear      - Line Bearing
    MADn       - Mean Ang Dist Rn
    NQPDAn     - NetQuantPD Ang Rn
    BtAn       - Betweenness Ang Rn
    TPBtAn     - TPBetweenness Ang Rn
    TPDn       - TPDestination Rn
    Lnkn       - Links Rn
    Lenn       - Length Rn
    AngDn      - Ang Dist Rn
    Wtn        - Weight Rn
    Jncn       - Junctions Rn
    Conn       - Connectivity Rn
    MGLAn      - MeanGeoLen Ang Rn
    MCFn       - Mean Crow Flight Rn
    DivAn      - Diversion Ratio Ang Rn
    HullAn     - Convex Hull Area Rn
    HullPn     - Convex Hull Perimeter Rn
    HullRn     - Convex Hull Max Radius Rn
    HullBn     - Convex Hull Bearing Rn
    HullSIn    - Convex Hull Shape Index Rn
Field names saved to out.names.csv
Reading features from temp4.shp
Reading features from temp4.shp
Progress: 100%
sDNA processing
Progress: 0.0%
  sDNA is running in 64-bit mode
  Building network and checking for tolerance errors...
  Running sDNA Integral calculation...
Progress: 99.9%
  Finalizing skim matrix
Progress: 100%
TIME 35.5 for outputskim;skimzone=CD_GEOCO_1
Writing out
Progress: 100%

out_skim.zip

rtsaboya commented 9 months ago

Thanks for taking the time to test it!

I see. It must be something with my setup, then. Is there any way I can narrow the problem down? Could it be my Python version, or something like that?

JamesParrott commented 9 months ago

No probs. Which version of sDNA is this from, and which environment are you running sDNA in (e.g. Arc or QGIS?), or which command are you entering at the command line to produce the error?

rtsaboya commented 9 months ago

sDNA: v.4.1.1 (installed over a previous version that needed a license, if I remember correctly) sDNA plugin (QGIS): 4.1.0 QGIS: 3.28.12-Firenze Python: 3.9.5

I'm using the the dialog box from the Skim Matrix of the Toolbox. The log it generates is as follows:

QGIS version: 3.28.12-Firenze
QGIS code revision: dcc633b2de
Qt version: 5.15.3
Python version: 3.9.5
GDAL version: 3.7.2
GEOS version: 3.12.0-CAPI-1.18.0
PROJ version: Rel. 9.3.0, September 1st, 2023
PDAL version: 2.5.5 (git-version: 24f78d)
Algorithm started at: 2024-02-28T14:29:07
Algorithm 'Skim Matrix' starting…
Input parameters:
{ 'advanced' : '', 'analmet' : 0, 'custommetric' : '', 'destweight' : '', 'disable' : '', 'end_gs' : '', 'input' : 'F:/Downloads/temp/temp4.shp', 'odfile' : '', 'oneway' : '', 'origweight' : '', 'output' : 'TEMPORARY_OUTPUT', 'skimdestzone' : 'CD_GEOCO_1', 'skimorigzone' : 'CD_GEOCO_1', 'start_gs' : '', 'weighting' : 0, 'zonefiles' : '' }

**********************************************************************
WARNING: sDNA ignores your selection and will process the entire layer
**********************************************************************
Converting input layer to shapefile: C:\Users\renato\AppData\Local\Temp\tmpiesv_5fc.shp
ERROR: Output file is not shp or csv: C:/Users/renato/AppData/Local/Temp/processing_ZyOgFt/7f0aacc51a634f37bb899fd1703e5373/output.file
Traceback (most recent call last):
File "C:\Users/renato/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\sdna\sdna_plugin_algorithm.py", line 212, in processAlgorithm
syntax = self.extract_syntax(args, context, feedback, source_crs)
File "C:\Users/renato/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\sdna\sdna_plugin_algorithm.py", line 279, in extract_syntax
raise Exception(message)
Exception: Output file is not shp or csv: C:/Users/renato/AppData/Local/Temp/processing_ZyOgFt/7f0aacc51a634f37bb899fd1703e5373/output.file

Execution failed after 0.51 seconds

Loading resulting layers
Algorithm 'Skim Matrix' finished

rtsaboya commented 9 months ago

And this is the error message in Log Messages > Processing

2024-02-28T14:29:07     CRITICAL    Traceback (most recent call last):
              File "C:\Users/renato/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\sdna\sdna_plugin_algorithm.py", line 212, in processAlgorithm
              syntax = self.extract_syntax(args, context, feedback, source_crs)
              File "C:\Users/renato/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\sdna\sdna_plugin_algorithm.py", line 279, in extract_syntax
              raise Exception(message)
             Exception: Output file is not shp or csv: C:/Users/renato/AppData/Local/Temp/processing_ZyOgFt/7f0aacc51a634f37bb899fd1703e5373/output.file

JamesParrott commented 9 months ago

Nice one - cheers. That's exactly what we want.

JamesParrott commented 9 months ago

Does the plug in pick the name output.file, or is it possible to choose something else, or override it, to something ending in .shp or .csv? For skim, isn't this the matrix file too, so it ought to be a .csv, to match:

"output","Output Skim Matrix File","OutFile","csv"

I think the code below will always require the output to file to end in .shp or .csv, for QGIS specific reasons.
Unless it's processing the Skim interface at that point, not the user's args?

        for name,path in syntax["outputs"].items():
            if path:
                _, file_extension = os.path.splitext(path.lower())
                if file_extension not in [".shp", ".csv"]:
                    # as it stands we can't force qgis to use shapefile as output, so complain and exit here if it's anything else
                    message = "Output file is not shp or csv: "+path
                    feedback.setProgressText("ERROR: "+message)
                    QgsMessageLog.logMessage("ERROR: "+message, "SDNA")
                    raise Exception(message)

https://github.com/fiftysevendegreesofrad/sdna_qgis3_plugin/blob/c755d83f4e63ae76f7608435f0db8c87681c0651/sdna_plugin_algorithm.py#L271

rtsaboya commented 9 months ago

humm... I'm not sure, as I don't know the workings of the plugin that well...

But it looks like this comment is explaining the critical point:

# as it stands we can't force qgis to use shapefile as output, so complain and exit here if it's anything else

I will have to check if there is a workaround inside QGIS, or the reason why it is not saving the temporary file in shp or csv.

Thanks!

fiftysevendegreesofrad commented 9 months ago

Hi, sorry for being late to the party here!

Skim matrix wants to write a csv file. I think you are letting QGIS automatically pick a temporary output filename, and the temporary name QGIS chooses doesn't end in .csv.

Hopefully if you pick an output filename for the skim matrix that ends in .csv then it will work?

rtsaboya commented 9 months ago

Yes, that did the trick. Just to be clear, what I did was:

In the last field in the Skim Matrix dialog box (Output Skim Matrix File), I clicked on the button on the right with 3 dots.
Save to File...
Chose a folder and a file name for the output, using "csv" as the file extension.

Thanks to both of you!

fiftysevendegreesofrad / sdna_plus

Exception: Output file is not shp or csv #27