search

ShervanGharari / EASYMORE

EASYMORE; EArth SYstem MOdeling REmapper
GNU General Public License v3.0
24 stars 21 forks source link

`NaN` values results if `target_shp_ID` is assigned to string values #74

Closed kasra-keshavarz closed 9 months ago

kasra-keshavarz commented 10 months ago

If values of the target_shp_ID column within the target_shp is of type string, then the resulting remapped_var_ID values are NaNs.

MCVE

import pandas as pd
import geopandas as gpd
import xarray as xr

import requests
import re
import json

# ArcGIS REST API endpoint for the specific ATS layer
alta_ats = "https://geospatial-sit.alberta.ca/titan/rest/services/base/alberta_township_system/MapServer/7/query"
# list of townships needed
township_list = [
    'T001R08W4',
    'T001R12W4',
    'T001R13W4',
    'T001R18W4',
    'T001R19W4',
    'T001R20W4',
    'T001R22W4',
    'T001R23W4',
    'T002R11W4',
    'T002R12W4',
    'T002R13W4',
    'T002R14W4',
    'T002R15W4',
    'T002R16W4',
    'T002R17W4',
    'T002R18W4',
    'T002R21W4',
    'T002R23W4'
]
# list of township IDs based on the `PID` column of the source ATS data
ids_list = [''.join(re.split("[TRW]", ats)[::-1]) for ats in township_list]
# string of list items
list_str = ', '.join([f"'{item}'" for item in ids_list])
# creating a custrom SQL query statement to find `ids_list` values in the `PID` column
sql_query = f"FILE_NAME IN ({list_str})"
# Specifying query parameters
params_ag = {
    'where': sql_query,
    'outFields': '*',
    'returnGeometry': 'true',
    'f': 'geojson',  # Output format as GeoJSON
}

# Make the request to the ArcGIS REST API
response_ag = requests.get(alta_ats, params=params_ag)

# Check if the request was successful
if response_ag.status_code == 200:
    # Load GeoJSON data into a GeoDataFrame
    gdf_ag = gpd.read_file(response_ag.text)
else:
    print(f"Error: {response_ag.status_code} - {response_ag.text}")

The Shapefile looks like the following:

# printing the GeoDataFrame
print(gdf_ag)
       PID FILE_NAME     TRM  M  RGE  TWP            DESCRIPTOR  OBJECTID  \
0   419001    419001  001194  4   19    1  TWP-001 RGE-19 MER-4       158   
1   413002    413002  002134  4   13    2  TWP-002 RGE-13 MER-4       370   
2   411002    411002  002114  4   11    2  TWP-002 RGE-11 MER-4      1634   
3   416002    416002  002164  4   16    2  TWP-002 RGE-16 MER-4      1959   
4   418002    418002  002184  4   18    2  TWP-002 RGE-18 MER-4      2002   
5   418001    418001  001184  4   18    1  TWP-001 RGE-18 MER-4      2752   
6   415002    415002  002154  4   15    2  TWP-002 RGE-15 MER-4      2808   
7   414002    414002  002144  4   14    2  TWP-002 RGE-14 MER-4      3741   
8   412001    412001  001124  4   12    1  TWP-001 RGE-12 MER-4      3757   
9   420001    420001  001204  4   20    1  TWP-001 RGE-20 MER-4      3947   
10  421002    421002  002214  4   21    2  TWP-002 RGE-21 MER-4      4306   
11  423001    423001  001234  4   23    1  TWP-001 RGE-23 MER-4      4948   
12  417002    417002  002174  4   17    2  TWP-002 RGE-17 MER-4      5068   
13  412002    412002  002124  4   12    2  TWP-002 RGE-12 MER-4      5425   
14  423002    423002  002234  4   23    2  TWP-002 RGE-23 MER-4      6052   
15  413001    413001  001134  4   13    1  TWP-001 RGE-13 MER-4      6066   
16  408001    408001  001084  4    8    1  TWP-001 RGE-08 MER-4      6695   
17  422001    422001  001224  4   22    1  TWP-001 RGE-22 MER-4      6959   

    SHAPE.STArea()  SHAPE.STLength()  \
0         0.011785          0.443881   
1         0.011649          0.441466   
2         0.011654          0.441594   
3         0.011660          0.441703   
4         0.011630          0.441222   
5         0.011703          0.442220   
6         0.011661          0.441647   
7         0.011678          0.442181   
8         0.011959          0.445959   
9         0.011701          0.441603   
10        0.011663          0.441734   
11        0.011728          0.442683   
12        0.011669          0.441912   
13        0.011657          0.441644   
14        0.011641          0.441307   
15        0.011881          0.444818   
16        0.011854          0.444674   
17        0.011764          0.443244   

                                             geometry  
0   POLYGON ((-112.41021 48.99883, -112.41021 49.0...  
1   POLYGON ((-111.62536 49.17359, -111.62572 49.1...  
2   POLYGON ((-111.37926 49.17368, -111.38062 49.1...  
3   POLYGON ((-112.00954 49.17366, -112.01234 49.1...  
4   POLYGON ((-112.33750 49.17373, -112.34289 49.1...  
5   POLYGON ((-112.31545 49.08647, -112.32094 49.0...  
6   POLYGON ((-112.00954 49.08620, -112.00926 49.0...  
7   POLYGON ((-111.82152 49.17359, -111.82180 49.1...  
8   POLYGON ((-111.48086 49.08632, -111.48635 49.0...  
9   POLYGON ((-112.62693 49.08680, -112.63238 49.0...  
10  POLYGON ((-112.67709 49.08661, -112.67709 49.0...  
11  POLYGON ((-112.94430 48.99872, -112.94430 49.0...  
12  POLYGON ((-112.25877 49.17370, -112.25974 49.1...  
13  POLYGON ((-111.49763 49.17367, -111.50238 49.1...  
14  POLYGON ((-113.07739 49.08641, -113.07191 49.0...  
15  POLYGON ((-111.69200 49.08629, -111.69748 49.0...  
16  POLYGON ((-111.07417 48.99769, -111.06868 48.9...  
17  POLYGON ((-112.81625 48.99854, -112.81076 48.9...

And the easymore's setup is as follows:

# using easymore to create aerial average values
from easymore import Easymore

easymore_conf = {
    'case_name': 'AB-townships-data',
    'target_shp': '/path/to/results/townships.shp', 
    'target_shp_ID': 'DESCRIPTOR',
    'temp_dir': '/path/to/temp_dir/', # only available locally
    'source_nc': '/path/to/met_vars.nc',
    'var_names': ['var1', 'var2'],
    'var_lon': 'lon',
    'var_lat': 'lat',
    'var_time': 'time',
    'remapped_var_id': 'DESCRIPTOR',
    'remapped_dim_id': 'DESCRIPTOR',
    'output_dir': '/path/to/township-ag/', # only available locally
}

remap_exp = Easymore.from_dict(easymore_conf)

Which runs successfully with:

remap_exp.nc_remapper()

The version of easymore is 2.0.0 likely the latest commit of the develop_2.0.0 branch.

ShervanGharari commented 10 months ago

This issue is "kind of" taken care of with easymore version 2.0.0. easymore can output the shapefile attribute as nc file in accordance with the provided file to carry all information, except geometry, from the shapefile in a format that can be appended to the remapped files based on need. This includes strings, names, etc. The order of attributes will be exactly the same as the remapped file with the same order. In your case, you can just leave out target_shp_ID. All the fields of the target shapefile will be passed to a file called self.temp_dir+self.case_name+'_attributes.nc'. This is also helpful in case a user forgets to specify the target_shp_ID, they can look at the attribute file always. For now time target_shp_ID should be always int with unique values. I can add a check for this.

ShervanGharari commented 10 months ago

In addition there should be a hash in remapping file, attribute file and remapped file that assure the what files are used for remapping or are the result of remapping.