[BUG]: geojson2coco error while converting AOI5_Khartoum (Spacenet Building Dataset)

ashnair1 commented 4 years ago

Summary of the bug

I wanted to convert the AOI 5 Khartoum dataset from geojson to the coco format. However I'm unable to do so due to certain dataframes having polygon fields as Nan in the affine_transform_gdf function.

Steps to reproduce the bug

The following is the code I used to do the conversion:

import solaris as sol 
import os
import json

data_dir = "/home/an1/Datasets/SpacenetBuildings/AOI_5_Khartoum_Train"

img_dir = os.path.join(data_dir, 'RGB-PanSharpen')
json_dir = os.path.join(data_dir, 'geojson', 'buildings')

sample_geojsons = sorted([os.path.join(json_dir, i) for i in os.listdir(json_dir)])
sample_images   = sorted([os.path.join(img_dir, i) for i in os.listdir(img_dir)])

coco_dict = sol.data.coco.geojson2coco(sample_images,
                                       sample_geojsons,
                                       matching_re=r'(AOI_\d{1}_[A-z]+_img\d+)',
                                       license_dict={'CC-BY 4.0': 'https://creativecommons.org/licenses/by/4.0/'},
                                       verbose=0)

with open(os.path.basename(data_dir) + '.json', 'w') as sj: 
    json.dump(coco_dict, sj)

AOI_5_Khartoum dataset organisation is the same as the tar file. I haven't changed it after untarring it.

Buggy behavior and/or error message

Please describe the buggy behavior and/or paste output here.

/home/an1/miniconda3/envs/solaris/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/an1/miniconda3/envs/solaris/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/an1/miniconda3/envs/solaris/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/an1/miniconda3/envs/solaris/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/an1/miniconda3/envs/solaris/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/an1/miniconda3/envs/solaris/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
  6%|██████▌                                                                                                         | 59/1012 [00:03<01:04, 14.80it/s]
Traceback (most recent call last):
  File "spacenet2coco.py", line 32, in <module>
    verbose=0)
  File "/home/an1/miniconda3/envs/solaris/lib/python3.7/site-packages/solaris/data/coco.py", line 229, in geojson2coco
    'image_fname'].values[0])
  File "/home/an1/miniconda3/envs/solaris/lib/python3.7/site-packages/solaris/vector/polygon.py", line 250, in geojson_to_px_gdf
    geom_col=geom_col)
  File "/home/an1/miniconda3/envs/solaris/lib/python3.7/site-packages/solaris/vector/polygon.py", line 127, in affine_transform_gdf
    gdf['geometry'] = gdf['geometry'].apply(shapely.wkt.loads)
  File "/home/an1/miniconda3/envs/solaris/lib/python3.7/site-packages/geopandas/geoseries.py", line 319, in apply
    result = super().apply(func, args=args, **kwargs)
  File "/home/an1/miniconda3/envs/solaris/lib/python3.7/site-packages/pandas/core/series.py", line 4200, in apply
    mapped = lib.map_infer(values, f, convert=convert_dtype)
  File "pandas/_libs/lib.pyx", line 2388, in pandas._libs.lib.map_infer
  File "/home/an1/miniconda3/envs/solaris/lib/python3.7/site-packages/shapely/wkt.py", line 10, in loads
    return geos.WKTReader(geos.lgeos).read(data)
  File "/home/an1/miniconda3/envs/solaris/lib/python3.7/site-packages/shapely/geos.py", line 279, in read
    raise TypeError("Only str is accepted.")

Environment information

OS: CentOS Linux 7
solaris version: 0.3.0
python version: 3.7.8

DiogoFerreira99 commented 2 years ago

I have a similar problem did you find a solution?

ashnair1 commented 2 years ago

Not exactly. I modified an old script of mine and got it to work. You can find the old script here. To be clear, the solution is incredibly hacky and I haven't looked at it for a couple of years so it might not be what you want.

rohitdileep commented 2 years ago

Pls let me know if you had solved the issue as i am also facing the same issue.

CosmiQ / solaris