Attempting to access remote EPT dataset using HTTP returns empty string error message

WY-CGhilardi commented 1 year ago

What is the bug or the crash?

I have an EPT dataset built and stored on a private S3 bucket which I would like to view using QGIS 3.32.2 (full version block below) I have refreshed/current AWS credentials stored in $USERPROFILE/.aws/credentials

Attempting to add the dataset via the Layer->Add layer -> Add Point Cloud Layer menu

This returns a top banner of:

Invalid Data Source: https://somebucket.s3.us-west-2.amazonaws.com/someprefix/ept.json is not a valid or recognized data source, error: ""

Under Log Messages -> Network the resulting log is

2023-09-06T08:03:45 WARNING Error transferring https://somebucket.s3.us-west-2.amazonaws.com/someprefix/ept.json - server replied: Forbidden

Using the F12 logger, the request and response look like:

{
  "Bytes Received": 255,
  "Bytes Total": 255,
  "Cache (control)": "Load from cache if available, otherwise load from network",
  "Cache (save)": "Can store result in cache",
  "Headers": {
    "User-Agent": "Mozilla/5.0 QGIS/33202/Windows 10 Version 2009"
  },
  "Initiator": "unknown",
  "Operation": "GET",
  "Replies": 2,
  "Reply": {
    "Cache (result)": "Read from network",
    "Error": "Error transferring https://somebucket.s3.us-west-2.amazonaws.com/someprefix/ept.json - server replied: Forbidden",
    "Error Code": "201",
    "Headers": {
      "Content-Type": "application/xml",
      "Date": "Wed, 06 Sep 2023 15:23:26 GMT",
      "Server": "AmazonS3",
      "Transfer-Encoding": "chunked",
      "x-amz-id-2": "RekkBAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGCJjxA==",
      "x-amz-request-id": "9NKQNFG93KFR77T4"
    },
    "Status": "403"
  },
  "Thread": "0x000002cfd9fceb10",
  "Total time (ms)": 3,
  "URL": "https://somebucket.s3.us-west-2.amazonaws.com/someprefix/ept.json"
}

I also tried manually loading the relevant credential parts into environment variables, but that returns the same result described above

#from python console
os.environ['AWS_DEFAULT_REGION'] = 'us-west-2'
os.environ['AWS_ACCESS_KEY_ID'] = 'ABCDEFGHIJKLMNO'
os.environ['AWS_SECRET_ACCESS_KEY'] = 'ABCDEFGHILJKLMNO'
os.environ['AWS_SESSION_TOKEN'] = 'THISISAVERYLONGSTRING'

I am able to successfully hit public facing buckets using the same syntax successfully.

Using https://s3-us-west-2.amazonaws.com/usgs-lidar-public/AK_Kenai_2008/ept.json (link from ~12:57 in this lutra webinar)

F12 logger Request/Reponse from public ept

``` { "Bytes Received": 0, "Bytes Total": 2436, "Cache (control)": "Load from cache if available, otherwise load from network", "Cache (save)": "Can store result in cache", "Headers": { "User-Agent": "Mozilla/5.0 QGIS/33202/Windows 10 Version 2009" }, "Initiator": "unknown", "Operation": "GET", "Replies": 2, "Reply": { "Cache (result)": "Used entry from cache", "Headers": { "Accept-Ranges": "bytes", "Content-Length": "2436", "Content-Type": "application/json", "Date": "Thu, 07 Sep 2023 19:22:16 GMT", "ETag": "\"ad015e102f25819593d4f81957001488\"", "Last-Modified": "Thu, 05 Aug 2021 20:36:09 GMT", "Server": "AmazonS3", "x-amz-id-2": "EWWjvhcOpvt4hwxdegdh8/8rrEU090fqvhUQxq3u2Uimm5cVFR1fTnFXyqxDJ+QL2ZJtXZtKGn0=", "x-amz-request-id": "A6X3HJ4V1HFQ333C" }, "Status": "200" }, "Thread": "0x0000020754b86a30", "Total time (ms)": 1, "URL": "https://s3-us-west-2.amazonaws.com/usgs-lidar-public/AK_Kenai_2008/ept.json" } ```

I can also successfully load a raster object from that same private bucket (different prefix) but only when using the S3 data provider

Given that I can access raster objects from the bucket successfully, I know QGIS is finding the credentials properly. What I don't understand is where they get lost when using the Point Cloud datasource provider. Is this just something that is not currently supported with the tool? At the very least, I think it would be helpful to have the actual error (forbidden) percolate up to the top level banner rather than returning an empty string

Steps to reproduce the issue

From the top menu, select Layer->Add layer -> Add Point Cloud Layer menu. Select HTTP radio button and input an HTTP endpoint for a ept.json hosted on S3

Versions

Full version info

QGIS version 3.32.2-Lima QGIS code revision c0b8833964 Qt version 5.15.3 Python version 3.9.5 GDAL/OGR version 3.7.1 PROJ version 9.2.1 EPSG Registry database version v10.088 (2023-05-13) GEOS version 3.12.0-CAPI-1.18.0 SQLite version 3.41.1 PDAL version 2.5.5 PostgreSQL client version 15.2 SpatiaLite version 5.1.0 QWT version 6.1.6 QScintilla2 version 2.13.1 OS version Windows 10 Version 2009 Active Python plugins FeatureGridCreator 3.0.0 GroupStats 2.2.7 mmqgis 2021.9.10 qgsAzimuth 0.9.16 shapetools 3.4.14 db_manager 0.1.20 grassprovider 2.12.99 MetaSearch 0.3.6 processing 2.12.99

Providers

Available QGIS Data Provider Plugins OGC API - Features data provider WFS data provider ArcGIS Feature Service data provider ArcGIS Map Service data provider ArcGIS Vector Tile Service data provider COPC point cloud data provider Delimited text data provider EPT point cloud data provider GDAL data provider GPS eXchange format provider GRASS 8 vector provider GRASS 8 raster provider SAP HANA spatial data provider MBTile Vector Tiles data provider MDAL provider Memory provider Mesh memory provider MSSQL spatial data provider OGR data provider Oracle data provider PDAL point cloud data provider PostgreSQL/PostGIS data provider Postgres raster provider SpatiaLite data provider Vector tile provider Virtual layer data provider Virtual Raster data provider Virtual point cloud data provider VTPK Vector Tiles data provider OGC Web Coverage Service version 1.0/1.1 data provider OGC Web Map Service version 1.3 data provider XYZ Vector Tiles data provider Available QGIS Authentication Method Plugins API Header AWS S3 Basic authentication ESRI token Identity certificate authentication MapTiler HMAC-SHA256 OAuth2 authentication PKI PKCS#12 authentication PKI paths authentication Available Qt Database Plugins QOCISPATIAL QOCISPATIAL8 QSPATIALITE QSQLITE QMARIADB QMYSQL QMYSQL3 QODBC QODBC3 QPSQL QPSQL7

Supported QGIS version

[X] I'm running a supported QGIS version according to the roadmap.

New profile

[X] I tried with a new QGIS profile

Additional context

No response

nirvn commented 11 months ago

@WY-CGhilardi , setting environment variables via python console won't impact the connection request here.

I have not tried as I do not have an AWS account but I'm pretty sure what you need to do here is add an authentication method (API header) with the AWS_* variables you listed above.

WY-CGhilardi commented 11 months ago

I can confirm that I get the same behavior with adding an API header authentication item. It is strange but, even with the API header set up, I do not see it getting passed through to the request?

From F12 tab:

github-actions[bot] commented 10 months ago

The QGIS project highly values your report and would love to see it addressed. However, this issue has been left in feedback mode for the last 14 days and is being automatically marked as "stale". If you would like to continue with this issue, please provide any missing information or answer any open questions. If you could resolve the issue yourself meanwhile, please leave a note for future readers with the same problem and close the issue. In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this issue. If there is no further activity on this issue, it will be closed in a week.

WY-CGhilardi commented 10 months ago

@nirvn Could we remove the feedback label on this? I responded above using your suggestion. I don't want the issue closed by the bot.

qgis / QGIS