search

databrickslabs / blueprint

Baseline for Databricks Labs projects written in Python
https://pypi.org/project/databricks-labs-blueprint/
Other
19 stars 8 forks source link

Fix `.as_uri()` and `.absolute()` implementations for `WorkspacePath` #126

Closed asnare closed 3 months ago

asnare commented 3 months ago

This PR is stacked on #122 and fixes some things that the integration tests expect but weren't covered by unit tests:

Both underlying implementations depend on PathLib internals that changed across python versions.

With this PR applied, some integrations still fail due to missing globbing support, which is implemented as #125.

github-actions[bot] commented 3 months ago

❌ 16/18 passed, 2 failed, 2 skipped, 29s total

❌ test_open_text_io: AttributeError: 'object' object has no attribute 'parse_parts' (1.308s) ``` AttributeError: 'object' object has no attribute 'parse_parts' 17:49 DEBUG [databricks.sdk] Loaded from environment 17:49 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred 17:49 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred 17:49 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service 17:49 INFO [databricks.sdk] Using Databricks Metadata Service authentication [gw6] linux -- Python 3.10.14 /home/runner/work/blueprint/blueprint/.venv/bin/python 17:49 DEBUG [databricks.sdk] Loaded from environment 17:49 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred 17:49 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred 17:49 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service 17:49 INFO [databricks.sdk] Using Databricks Metadata Service authentication 17:49 DEBUG [databricks.sdk] GET /api/2.0/preview/scim/v2/Me < 200 OK < { < "active": true, < "displayName": "labs-runtime-identity", < "emails": [ < { < "primary": true, < "type": "work", < "value": "**REDACTED**" < } < ], < "externalId": "d0f9bd2c-5651-45fd-b648-12a3fc6375c4", < "groups": [ < { < "$ref": "Groups/300667344111082", < "display": "labs.scope.runtime", < "type": "direct", < "value": "**REDACTED**" < } < ], < "id": "4643477475987733", < "name": { < "givenName": "labs-runtime-identity" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:User", < "... (1 additional elements)" < ], < "userName": "4106dc97-a963-48f0-a079-a578238959a6" < } 17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/mkdirs > { > "path": "/Users/4106dc97-a963-48f0-a079-a578238959a6/nCQeJ3P03qkoN4m4/a/b/c" > } < 200 OK < {} 17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/import > [raw stream] < 200 OK < { < "object_id": 3200206029686177 < } 17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/import > [raw stream] < 400 Bad Request < { < "error_code": "RESOURCE_ALREADY_EXISTS", < "message": "Path (/Users/4106dc97-a963-48f0-a079-a578238959a6/nCQeJ3P03qkoN4m4/a/b/c/hello.txt) already exis... (3 more bytes)" < } 17:49 DEBUG [databricks.sdk] GET /api/2.0/workspace/export?path=/Users/4106dc97-a963-48f0-a079-a578238959a6/nCQeJ3P03qkoN4m4/a/b/c/hello.txt&direct_download=true&format=AUTO < 200 OK < [raw stream] 17:49 DEBUG [databricks.sdk] Loaded from environment 17:49 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred 17:49 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred 17:49 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service 17:49 INFO [databricks.sdk] Using Databricks Metadata Service authentication 17:49 DEBUG [databricks.sdk] GET /api/2.0/preview/scim/v2/Me < 200 OK < { < "active": true, < "displayName": "labs-runtime-identity", < "emails": [ < { < "primary": true, < "type": "work", < "value": "**REDACTED**" < } < ], < "externalId": "d0f9bd2c-5651-45fd-b648-12a3fc6375c4", < "groups": [ < { < "$ref": "Groups/300667344111082", < "display": "labs.scope.runtime", < "type": "direct", < "value": "**REDACTED**" < } < ], < "id": "4643477475987733", < "name": { < "givenName": "labs-runtime-identity" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:User", < "... (1 additional elements)" < ], < "userName": "4106dc97-a963-48f0-a079-a578238959a6" < } 17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/mkdirs > { > "path": "/Users/4106dc97-a963-48f0-a079-a578238959a6/nCQeJ3P03qkoN4m4/a/b/c" > } < 200 OK < {} 17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/import > [raw stream] < 200 OK < { < "object_id": 3200206029686177 < } 17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/import > [raw stream] < 400 Bad Request < { < "error_code": "RESOURCE_ALREADY_EXISTS", < "message": "Path (/Users/4106dc97-a963-48f0-a079-a578238959a6/nCQeJ3P03qkoN4m4/a/b/c/hello.txt) already exis... (3 more bytes)" < } 17:49 DEBUG [databricks.sdk] GET /api/2.0/workspace/export?path=/Users/4106dc97-a963-48f0-a079-a578238959a6/nCQeJ3P03qkoN4m4/a/b/c/hello.txt&direct_download=true&format=AUTO < 200 OK < [raw stream] [gw6] linux -- Python 3.10.14 /home/runner/work/blueprint/blueprint/.venv/bin/python ```
❌ test_file_and_notebook_in_same_folder_with_different_suffixes: AttributeError: 'object' object has no attribute 'parse_parts' (897ms) ``` AttributeError: 'object' object has no attribute 'parse_parts' 17:49 DEBUG [databricks.sdk] Loaded from environment 17:49 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred 17:49 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred 17:49 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service 17:49 INFO [databricks.sdk] Using Databricks Metadata Service authentication [gw7] linux -- Python 3.10.14 /home/runner/work/blueprint/blueprint/.venv/bin/python 17:49 DEBUG [databricks.sdk] Loaded from environment 17:49 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred 17:49 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred 17:49 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service 17:49 INFO [databricks.sdk] Using Databricks Metadata Service authentication 17:49 DEBUG [databricks.sdk] GET /api/2.0/preview/scim/v2/Me < 200 OK < { < "active": true, < "displayName": "labs-runtime-identity", < "emails": [ < { < "primary": true, < "type": "work", < "value": "**REDACTED**" < } < ], < "externalId": "d0f9bd2c-5651-45fd-b648-12a3fc6375c4", < "groups": [ < { < "$ref": "Groups/300667344111082", < "display": "labs.scope.runtime", < "type": "direct", < "value": "**REDACTED**" < } < ], < "id": "4643477475987733", < "name": { < "givenName": "labs-runtime-identity" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:User", < "... (1 additional elements)" < ], < "userName": "4106dc97-a963-48f0-a079-a578238959a6" < } 17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/mkdirs > { > "path": "/Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg" > } < 200 OK < {} 17:49 DEBUG [databricks.labs.blueprint.tests] added directory fixture: /Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg 17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/import > [raw stream] < 200 OK < { < "object_id": 3200206029686206 < } 17:49 DEBUG [databricks.labs.blueprint.tests] added notebook fixture: /Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg/b 17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/import > [raw stream] < 200 OK < { < "object_id": 3200206029686208 < } 17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/import > [raw stream] < 400 Bad Request < { < "error_code": "RESOURCE_ALREADY_EXISTS", < "message": "Path (/Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg/a.txt) already exists." < } 17:49 DEBUG [databricks.sdk] Loaded from environment 17:49 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred 17:49 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred 17:49 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service 17:49 INFO [databricks.sdk] Using Databricks Metadata Service authentication 17:49 DEBUG [databricks.sdk] GET /api/2.0/preview/scim/v2/Me < 200 OK < { < "active": true, < "displayName": "labs-runtime-identity", < "emails": [ < { < "primary": true, < "type": "work", < "value": "**REDACTED**" < } < ], < "externalId": "d0f9bd2c-5651-45fd-b648-12a3fc6375c4", < "groups": [ < { < "$ref": "Groups/300667344111082", < "display": "labs.scope.runtime", < "type": "direct", < "value": "**REDACTED**" < } < ], < "id": "4643477475987733", < "name": { < "givenName": "labs-runtime-identity" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:User", < "... (1 additional elements)" < ], < "userName": "4106dc97-a963-48f0-a079-a578238959a6" < } 17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/mkdirs > { > "path": "/Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg" > } < 200 OK < {} 17:49 DEBUG [databricks.labs.blueprint.tests] added directory fixture: /Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg 17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/import > [raw stream] < 200 OK < { < "object_id": 3200206029686206 < } 17:49 DEBUG [databricks.labs.blueprint.tests] added notebook fixture: /Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg/b 17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/import > [raw stream] < 200 OK < { < "object_id": 3200206029686208 < } 17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/import > [raw stream] < 400 Bad Request < { < "error_code": "RESOURCE_ALREADY_EXISTS", < "message": "Path (/Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg/a.txt) already exists." < } 17:49 DEBUG [databricks.labs.blueprint.tests] clearing 1 directory fixtures 17:49 DEBUG [databricks.labs.blueprint.tests] removing directory fixture: /Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg 17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/delete > { > "path": "/Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg", > "recursive": true > } < 200 OK < {} 17:49 DEBUG [databricks.labs.blueprint.tests] clearing 1 notebook fixtures 17:49 DEBUG [databricks.labs.blueprint.tests] removing notebook fixture: /Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg/b 17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/delete > { > "path": "/Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg/b" > } < 404 Not Found < { < "error_code": "RESOURCE_DOES_NOT_EXIST", < "message": "Path (/Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg/b) doesn't exist." < } 17:49 DEBUG [databricks.labs.blueprint.tests] ignoring error while notebook /Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg/b teardown: Path (/Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg/b) doesn't exist. [gw7] linux -- Python 3.10.14 /home/runner/work/blueprint/blueprint/.venv/bin/python ```

Running from acceptance #178

asnare commented 3 months ago

Rebased (and superseded) by #127.