search

OSGeo / grass

GRASS GIS - free and open-source geospatial processing engine
https://grass.osgeo.org
Other
854 stars 310 forks source link

[Bug] v.in.wfs fails when special characters exist in the wfs url #4275

Open karunakar2 opened 2 months ago

karunakar2 commented 2 months ago

Describe the bug

The v.in.wfs throws an error "raise InvalidURL(f"URL can't contain control characters. {url!r} " when the url has special characters as a part of api validation.

To reproduce

the wfs provided by koordinates.co.nz has url with api embedded in the url

  1. Go to QGIS processing toolbox and open v.in.wfs (grass module)
  2. Set the koordinates wfs url (https://data.waikatodistrict.govt.nz/services;key=/wfs?
  3. run the algorithm
  4. See error in the log window -->

Expected behavior

WFS should work as usual if the urllib issue is fixed.

Screenshots

System description

Additional context

karunakar2 commented 2 months ago

owslib circumvents the issue

tmszi commented 2 months ago

Hello I tried your WFS URL example https://data.waikatodistrict.govt.nz/services;key=/wfs? with Python OWSLib, with default WFS version 1.0.0 or 1.1.0 but I got HTTP error.

from owslib.wfs import WebFeatureService
wfs = WebFeatureService(url="https://data.waikatodistrict.govt.nz/services;key=/wfs?")

Error message:

HTTPError: 404 Client Error: Not Found for url: https://data.waikatodistrict.govt.nz/services;key=/wfs?service=WFS&request=GetCapabilities&version=1.0.0

Could you provide functional public WFS URL example please?

karunakar2 commented 2 months ago

Hi Tomas, The API key has been removed for this case, but is the case where urllib needs an update. May have to get a temp key from the site at root.

On Sat, 7 Sept 2024, 19:31 Tomas Zigo, @.***> wrote:

Hello I tried your WFS URL example https://data.waikatodistrict.govt.nz/services;key=/wfs? with Python OWSLib, with default WFS version 1.0.0 or 1.1.0 but I got HTTP error.

from owslib.wfs import WebFeatureServicewfs = WebFeatureService(url="https://data.waikatodistrict.govt.nz/services;key=/wfs?")

Error message:

HTTPError: 404 Client Error: Not Found for url: https://data.waikatodistrict.govt.nz/services;key=/wfs?service=WFS&request=GetCapabilities&version=1.0.0

Could you provide functional public WFS URL example please?

— Reply to this email directly, view it on GitHub https://github.com/OSGeo/grass/issues/4275#issuecomment-2335103690, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALAMZ5U6MMVKKRPUEJLAC3DZVKTU5AVCNFSM6AAAAABNVOT2AGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZVGEYDGNRZGA . You are receiving this because you authored the thread.Message ID: @.***>

tmszi commented 2 months ago

I registered at page https://data.waikatodistrict.govt.nz/ and I tried download some data via WFS service successfully with GRASS GIS module v.in.wfs.

v.in.wfs command (Change URL <API_KEY> to your relevant service API KEY.):

v.in.wfs "url=https://data.waikatodistrict.govt.nz/services;key=<API_KEY>/wfs/?" output=imported_wfs_data name=data.waikatodistrict.govt.nz:layer-95697

Example:

GRASS epsg_2193/PERMANENT:~ > v.in.wfs "url=https://data.waikatodistrict.govt.nz/services;key=<API_KEY>/wfs/?" output=imported_wfs_data name=data.waikatodistrict.govt.nz:layer-95697
https://data.waikatodistrict.govt.nz/services;key=<API_KEY>/wfs/?REQUEST=GetFeature&SERVICE=WFS&VERSION=1.0.0&TYPENAME=data.waikatodistrict.govt.nz:layer-95697
Retrieving data...
Importing data...
No projection information available for layer <layer-95697>
Over-riding projection check
Check if OGR layer <layer-95697> contains polygons...
 100%
Creating attribute table for layer <layer-95697>...
Importing 1 features (OGR layer <layer-95697>)...
 100%
-----------------------------------------------------
Registering primitives...
-----------------------------------------------------
Cleaning polygons
-----------------------------------------------------
Breaking polygons...
Breaking polygons (pass 1: select break points)...
 100%
Breaking polygons (pass 2: break at selected points)...
 100%
-----------------------------------------------------
Removing duplicates...
 100%
-----------------------------------------------------
Breaking boundaries...
 100%
-----------------------------------------------------
Removing duplicates...
 100%
-----------------------------------------------------
Cleaning boundaries at nodes...
 100%
-----------------------------------------------------
Merging boundaries...
 100%
-----------------------------------------------------
Removing dangles...
 100%
-----------------------------------------------------
Building areas...
 100%
-----------------------------------------------------
Removing bridges...
 100%
-----------------------------------------------------
Registering primitives...
Building areas...
 100%
Attaching islands...
 100%
-----------------------------------------------------
Finding centroids for OGR layer <layer-95697>...
 100%
-----------------------------------------------------
Writing centroids...
 100%
-----------------------------------------------------
1 input polygons
Total area: 472316 (1 areas)
-----------------------------------------------------
Copying features...
 100%
Building topology for vector map <imported_wfs_data@PERMANENT>...
Registering primitives...
Building areas...
 100%
Attaching islands...
 100%
Attaching centroids...
 100%
Vector map <imported_wfs_data> imported from WFS.

Imported WFS vector map imported_wfs_data metadata:

GRASS epsg_2193/PERMANENT:~ > v.info imported_wfs_data
 +----------------------------------------------------------------------------+
 | Name:            imported_wfs_data                                         |
 | Mapset:          PERMANENT                                                 |
 | Project:         epsg_2193                                                 |
 | Database:        /home/tomas/grassdata                                     |
 | Title:                                                                     |
 | Map scale:       1:1                                                       |
 | Name of creator: tomas                                                     |
 | Organization:                                                              |
 | Source date:     Sun Sep 15 04:49:08 2024                                  |
 | Timestamp (first layer): none                                              |
 |----------------------------------------------------------------------------|
 | Map format:      native                                                    |
 |----------------------------------------------------------------------------|
 |   Type of map: vector (level: 2)                                           |
 |                                                                            |
 |   Number of points:       0               Number of centroids:  1          |
 |   Number of lines:        0               Number of boundaries: 1          |
 |   Number of areas:        1               Number of islands:    1          |
 |                                                                            |
 |   Map is 3D:              No                                               |
 |   Number of dblinks:      1                                                |
 |                                                                            |
 |   Projection: NZGD2000 / New Zealand Transverse Mercator 2000              |
 |                                                                            |
 |               N:  5825400.23772001    S:  5824481.42474173                 |
 |               E:  1794852.97836222    W:  1793671.01079498                 |
 |                                                                            |
 |   Digitization threshold: 0                                                |
 |   Comment:                                                                 |
 |                                                                            |
 +----------------------------------------------------------------------------+

Could you try this example (or with another data from this WFS service) with GRASS GIS directly, and confirm that importing data is successful, please?

karunakar2 commented 2 months ago

Will try again and update here. Any specific version of grass?

Thanks

On Sun, 15 Sept 2024, 15:00 Tomas Zigo, @.***> wrote:

I registered at page https://data.waikatodistrict.govt.nz/ and I tried download some data via WFS service successfully with GRASS GIS module v.in.wfs.

GRASS epsg_2193/PERMANENT:~ > v.in.wfs --o "url=https://data.waikatodistrict.govt.nz/services;key=/wfs/?" output=imported_wfs_data name=data.waikatodistrict.govt.nz:layer-95697https://data.waikatodistrict.govt.nz/services;key=/wfs/?REQUEST=GetFeature&SERVICE=WFS&VERSION=1.0.0&TYPENAME=data.waikatodistrict.govt.nz:layer-95697 Retrieving data... Importing data... No projection information available for layer Over-riding projection check Check if OGR layer contains polygons... 100% Creating attribute table for layer ... Importing 1 features (OGR layer )... 100%

Registering primitives...

Cleaning polygons

Breaking polygons... Breaking polygons (pass 1: select break points)... 100% Breaking polygons (pass 2: break at selected points)... 100%

Removing duplicates... 100%

Breaking boundaries... 100%

Removing duplicates... 100%

Cleaning boundaries at nodes... 100%

Merging boundaries... 100%

Removing dangles... 100%

Building areas... 100%

Removing bridges... 100%

Registering primitives... Building areas... 100% Attaching islands... 100%

Finding centroids for OGR layer ... 100%

Writing centroids... 100%

1 input polygons Total area: 472316 (1 areas)

Copying features... 100% Building topology for vector map @.***>... Registering primitives... Building areas... 100% Attaching islands... 100% Attaching centroids... 100% Vector map imported from WFS.

Imported WFS vector map metadata:

GRASS epsg_2193/PERMANENT:~ > v.info imported_wfs_data +----------------------------------------------------------------------------+ Name: imported_wfs_data Mapset: PERMANENT Project: epsg_2193 Database: /home/tomas/grassdata Title: Map scale: 1:1 Name of creator: tomas Organization: Source date: Sun Sep 15 04:49:08 2024 Timestamp (first layer): none
Map format: native
----------------------------------------------------------------------------
Type of map: vector (level: 2)
Number of points: 0 Number of centroids: 1
Number of lines: 0 Number of boundaries: 1
Number of areas: 1 Number of islands: 1
Map is 3D: No
Number of dblinks: 1
Projection: NZGD2000 / New Zealand Transverse Mercator 2000
N: 5825400.23772001 S: 5824481.42474173
E: 1794852.97836222 W: 1793671.01079498
Digitization threshold: 0
Comment:

+----------------------------------------------------------------------------+

Change URL to your relevant service API KEY.

Could you try this example (or with another data from this WFS service) with GRASS GIS directly, and confirm that importing data is successful, please?

— Reply to this email directly, view it on GitHub https://github.com/OSGeo/grass/issues/4275#issuecomment-2351325190, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALAMZ5VJQUHVIEQAS3ATTK3ZWTZ5VAVCNFSM6AAAAABNVOT2AGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNJRGMZDKMJZGA . You are receiving this because you authored the thread.Message ID: @.***>

tmszi commented 2 months ago

Any specific version of grass?

I tested with 8.5.0dev version, but you can try last stable 8.4.0 or 8.3 version.