search

RTIInternational / ngiab-teehr

A repository for coupling TEEHR with Nextgen-In-A-Box (NGIAB) simulation output
MIT License
0 stars 2 forks source link

Local dask client interferes with pyspark #1

Open samlamont opened 1 month ago

samlamont commented 1 month ago

Somehow just starting a local dask client via 

from dask.distributed import Client

client = Client()

interferes with pyspark, seems like just in the USGS and NWM fetching. Probably in the _get_secondary_location_ids() function

        lcw_df = self.ev.location_crosswalks.query(
            filters={
                "column": "secondary_location_id",
                "operator": "like",
                "value": f"{prefix}-%"
            }
        ).to_pandas()

where nothing gets returned by the query and the missing table error is raised. If we do not initialize the client, it works fine.

samlamont commented 1 month ago

However, it seems to work fine on TEEHR-Hub v0.4-beta either way