TheDataStation / ver

Data Discovery Tools and Systems
MIT License
5 stars 9 forks source link

Add regex patterns for spatial columns with the corresponding tests #75

Closed luthfibalaka closed 3 months ago

luthfibalaka commented 4 months ago

For the tests, I added it to PreAnalyzerTest

snowgy commented 3 months ago

I tested this ddprofiler with this file.

83ca-t3if2.csv

The current implementation can't identify location as a geo-coordinate. Could you look into that? Thanks!

luthfibalaka commented 3 months ago

I tested this ddprofiler with this file.

83ca-t3if2.csv

The current implementation can't identify location as a geo-coordinate. Could you look into that? Thanks!

There is a space between "POINT" and the coordinates. I have updated the pattern to incorporate that.

snowgy commented 3 months ago

I tested this ddprofiler with this file. 83ca-t3if2.csv The current implementation can't identify location as a geo-coordinate. Could you look into that? Thanks!

There is a space between "POINT" and the coordinates. I have updated the pattern to incorporate that.

have you tested it though? I try to run the ddprofiler with the new regex, but it still does not work. I follow the steps in the quickstart guide and look at the output json profile. The output granularity is None.

luthfibalaka commented 3 months ago

have you tested it though? I try to run the ddprofiler with the new regex, but it still does not work. I follow the steps in the quickstart guide and look at the output json profile. The output granularity is None.

I tested it by providing the value directly (in a test file). I will try running Ver as in quickstart guide.

luthfibalaka commented 3 months ago

have you tested it though? I try to run the ddprofiler with the new regex, but it still does not work. I follow the steps in the quickstart guide and look at the output json profile. The output granularity is None.

I tested it by providing the value directly (in a test file). I will try running Ver as in quickstart guide.

@snowgy I rebuilt and run the profiler. The result is a correct label ("semanticType":{"granularity":"geoCoordinate","type":"SPATIAL"}}) for the location column. I attached the corresponding profile below.

quickstart.83ca-t3if2.csv.location.4093332935.json

snowgy commented 3 months ago

have you tested it though? I try to run the ddprofiler with the new regex, but it still does not work. I follow the steps in the quickstart guide and look at the output json profile. The output granularity is None.

I tested it by providing the value directly (in a test file). I will try running Ver as in quickstart guide.

@snowgy I rebuilt and run the profiler. The result is a correct label ("semanticType":{"granularity":"geoCoordinate","type":"SPATIAL"}}) for the location column. I attached the corresponding profile below.

quickstart.83ca-t3if2.csv.location.4093332935.json

oops. it turns out that I must rebuild the ddprofiler to make the updated spatial patterns effective.

snowgy commented 3 months ago

well done!