luca-moscatelli commented 3 months ago

Hi

I encounter an error when trying to get a collection that I previously loaded with the GeoDB client on Windows 10and i use the 1.0.5 geodb version installed from conda. I followed the commands provided in the GeoDB notebook, and the shapefile I loaded is also from the repository. also from browser http../{collecion_name} the json i receive is correct.

Can you help me troubleshoot this issue?

Thank you.

--------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[66], [line 3](vscode-notebook-cell:?execution_count=66&line=3)
      [1](vscode-notebook-cell:?execution_count=66&line=1) geodb.create_collection(collection='land1_use1',properties=collections['land_use1']['properties'],database=db)
      [2](vscode-notebook-cell:?execution_count=66&line=2) geodb.insert_into_collection(collection='land1_use1', values=gdf.iloc[:100,:],database=db)
----> [3](vscode-notebook-cell:?execution_count=66&line=3) geodb.get_collection(collection='land1_use1',database=db)

File c:\Users\LucaMoscatelli\Anaconda3\envs\xcube_geodb\lib\site-packages\xcube_geodb\core\geodb.py:1788, in GeoDBClient.get_collection(self, collection, query, database, limit, offset)
   [1785](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:1785) js = r.json()
   [1787](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:1787) if js:
-> [1788](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:1788)     return self._df_from_json(js, srid)
   [1789](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:1789) else:
   [1790](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:1790)     return DataFrame(columns=["Empty Result"])

File c:\Users\LucaMoscatelli\Anaconda3\envs\xcube_geodb\lib\site-packages\xcube_geodb\core\geodb.py:1910, in GeoDBClient._df_from_json(self, js, srid)
   [1907](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:1907) if js is None:
   [1908](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:1908)     return DataFrame()
-> [1910](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:1910) data = [self._load_geo(d) for d in js]
   [1912](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:1912) gpdf = gpd.GeoDataFrame(data)
   [1914](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:1914) if 'geometry' in gpdf:

File c:\Users\LucaMoscatelli\Anaconda3\envs\xcube_geodb\lib\site-packages\xcube_geodb\core\geodb.py:1910, in <listcomp>(.0)
   [1907](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:1907) if js is None:
   [1908](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:1908)     return DataFrame()
-> [1910](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:1910) data = [self._load_geo(d) for d in js]
...
    [318](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/shapely/io.py:318) # of array elements)
    [319](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/shapely/io.py:319) geometry = np.asarray(geometry, dtype=object)
--> [320](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/shapely/io.py:320) return lib.from_wkb(geometry, invalid_handler, **kwargs)

TypeError: Expected bytes or string, got dict

thomasstorm commented 3 months ago

Dear Luca,

have you deleted the collection in the meantime? I cannot find it in the database. Please, create it anew, so I can check what is wrong. Also, could you please tell me your geoDB username? You can get it using the client: geodb.whoami. Thanks!

Cheers, Thomas

luca-moscatelli commented 3 months ago

hi thanks for the answer, you don't find the db beacuse i use a local enviroment my geodb run in local with geodb rest api , i followed this guide, so i'm not connect to a public db, also i have some problem with the geodb.publish_gs() beacuse it calls thi url http://localhost:8080/api/v2/services/xcube_geoserv/databases/testDB1/collections from geoserver but the path dosen't exist

thomasstorm commented 3 months ago

Dear Luca, interesting, thanks! My first suggestion is to update your geoDB client - the current release version is 1.0.6. Let me know if that solves the first issue you have mentioned. As to the publishing to the GeoServer: This is not covered in the guide. You will need to configure and run your own installation of the xcube-hub server software, which is available from here. Let me know if you are successful or need further assistance. If setting up your own geoDB installation is not a hard requirement, there is also the possibility to use the public service on https://eurodatacube.com/.

thomasstorm commented 3 months ago

Dear Luca, is the issue still relevant, or can we close it?

Cheers, Thomas

luca-moscatelli commented 3 months ago

hi yes is still important, i build the image from xcube-hub but the docker image i built it has this error when is running in the contaniner :openapi_server | Traceback (most recent call last): openapi_server | File "<frozen runpy>", line 198, in _run_module_as_main openapi_server | File "<frozen runpy>", line 88, in _run_code openapi_server | File "/home/xcube/xcube_hub/__main__.py", line 2, in <module> openapi_server | from xcube_hub import service openapi_server | File "/home/xcube/xcube_hub/service.py", line 5, in <module> openapi_server | from connexion.decorators.validation import ParameterValidator, RequestBodyValidator openapi_server | ModuleNotFoundError: No module named 'connexion.decorators.validation' openapi_server exited with code 1

thomasstorm commented 3 months ago

Dear Luca,

it seems your environment is missing the connexion package.

luca-moscatelli commented 3 months ago

hi I updated the geodb to 1.0.6 but the problem remain when i do : get_collection,create_collection_if_not_exists, i have this :

[319](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/shapely/io.py:319) geometry = np.asarray(geometry, dtype=object)
--> [320](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/shapely/io.py:320) return lib.from_wkb(geometry, invalid_handler, **kwargs)
TypeError: Expected bytes or string, got dict

while getCollections works correctly

thomasstorm commented 3 months ago

Hi Luca,

I am not familiar with the error message, and I would have to take a look into your actual data. Feel free to send me your data and the exact steps you took to receive this error.

luca-moscatelli commented 2 months ago

sure

import geopandas as gpd
from dotenv import load_dotenv
from xcube_geodb.core.geodb import GeoDBClient
import pandas as pd

# Then create the client
geodb = GeoDBClient(gs_server_url="http://localhost",gs_server_port='8080')

db='testDB1'

# Set up the GeoDB client

# Load shapefile using geopandas
gdf = gpd.read_file('data/sample/land_use.shp')

collections = {
    "land_use": {
        "crs": 4326,
        "properties": {
            "RABA_PID": "float",
            "RABA_ID": "float",
            "D_OD": "date"
        }
    },
    "land_use1": {
        "crs": 4326,
        "properties": {
            "RABA_PID": "float",
            "RABA_ID": "float",
            "D_OD": "date"
        }
    },
      "land_use2": {
        "crs": 4326,
        "properties": {
             "index": "int",

        }
    }
}
gdf.columns
gdf.plot(column="RABA_ID", figsize=(15,15), cmap = 'jet')

print(geodb.create_collection_if_not_exists(collection='land1_use6',properties=collections['land1_use1']['properties'],database=db,crs=4326))
geodb.insert_into_collection(collection='land1_use6', values=gdf.iloc[:100,:],database=db)

output:

TypeError                                 Traceback (most recent call last)
Cell In[27], [line 1](vscode-notebook-cell:?execution_count=27&line=1)
----> [1](vscode-notebook-cell:?execution_count=27&line=1) print(geodb.create_collection_if_not_exists(collection='land1_use6',properties=collections['land1_use1']['properties'],database=db,crs=4326))
      [2](vscode-notebook-cell:?execution_count=27&line=2) geodb.insert_into_collection(collection='land1_use6', values=gdf.iloc[:100,:],database=db)

File c:\Users\LucaMoscatelli\Anaconda3\envs\xcube_geodb\lib\site-packages\xcube_geodb\core\geodb.py:653, in create_collection_if_not_exists(self, collection, properties, crs, database, **kwargs)
    [632](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:632) def create_collection_if_not_exists(self,
    [633](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:633)                                     collection: str,
    [634](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:634)                                     properties: Dict,
    [635](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:635)                                     crs: Union[int, str] = 4326,
    [636](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:636)                                     database: Optional[str] = None,
    [637](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:637)                                     **kwargs) -> Union[Dict, Message]:
    [638](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:638)     """
    [639](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:639)     Creates a collection only if the collection does not exist already.
    [640](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:640) 
    [641](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:641)     Args:
    [642](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:642)         collection (str): The name of the collection to be created
    [643](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:643)         properties (Dict): Properties to be added to the collection
    [644](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:644)         crs (int, str): projection
    [645](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:645)         database (str): The database the collection is to be created in [current database]
    [646](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:646)         kwargs: Placeholder for deprecated parameters
    [647](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:647) 
    [648](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:648)     Returns:
    [649](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/xcube_geodb/core/geodb.py:649)         Collection:  Collection info id operation succeeds
...
    [318](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/shapely/io.py:318) # of array elements)
    [319](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/shapely/io.py:319) geometry = np.asarray(geometry, dtype=object)
--> [320](file:///C:/Users/LucaMoscatelli/Anaconda3/envs/xcube_geodb/lib/site-packages/shapely/io.py:320) return lib.from_wkb(geometry, invalid_handler, **kwargs)

TypeError: Expected bytes or string, got dict

this code has the same error : `return lib.from_wkb(geometry, invalid_handler, **kwargs)`

geodb.get_collection_info(collection='land1_use6',database=db)
gf=geodb.get_collection(collection='land1_use6',database=db,limit=10)
print(gf)

the db view :

 geodb.insert_into_collection(collection='land1_use6', 
                             values=gdf,
                             database='testDB1'
                             )

TypeError Traceback (most recent call last) Cell In[8], line 1 ----> 1 geodb.insert_into_collection(collection='land_use4', 2 values=gdf, 3 database='testDB1' 4 )

File c:\Users\LucaMoscatelli\Anaconda3\envs\xcube_geodb\lib\site-packages\xcube_geodb\core\geodb.py:1467, in GeoDBClient.insert_into_collection(self, collection, values, upsert, crs, database, max_transfer_chunk_size) 1464 headers = None 1466 try: -> 1467 self._post(f'/{dn}', payload=js, headers=headers) 1468 except GeoDBError as e: 1469 return self._maybe_raise(e)

File c:\Users\LucaMoscatelli\Anaconda3\envs\xcube_geodb\lib\site-packages\xcube_geodb\core\geodb.py:474, in GeoDBClient._post(self, path, payload, params, headers, raise_for_status) 472 r = requests.post(self._get_full_url(path=path), data=payload, params=params, headers=common_headers) 473 else: --> 474 r = requests.post(self._get_full_url(path=path), json=payload, params=params, headers=common_headers) 475 if raise_for_status: 476 r.raise_for_status()

File c:\Users\LucaMoscatelli\Anaconda3\envs\xcube_geodb\lib\site-packages\requests\api.py:115, in post(url, data, json, kwargs) 103 def post(url, data=None, json=None, kwargs): 104 r"""Sends a POST request. ... 178 """ --> 179 raise TypeError(f'Object of type {o.class.name} ' 180 f'is not JSON serializable')

TypeError: Object of type Timestamp is not JSON serializable


i hope this comment can be useful to figure out what is my issue thanks for the help

xcube-dev / xcube-geodb

error geodb.get_collection #91

this code has the same error : `return lib.from_wkb(geometry, invalid_handler, **kwargs)`

xcube-dev / xcube-geodb

error geodb.get_collection #91

this code has the same error : return lib.from_wkb(geometry, invalid_handler, **kwargs)

this code has the same error : `return lib.from_wkb(geometry, invalid_handler, **kwargs)`