TheDataStation / ver

Data Discovery Tools and Systems
MIT License
6 stars 10 forks source link

Xsystem Analyzer and Label Analyzer did not work when enabled #69

Open snowgy opened 10 months ago

snowgy commented 10 months ago

I was trying to use xsystem analyzer and label analyzer to label spatial-temporal attributes. I modified profile_schema.yml to enable both analyzers. However, when I ran ddprofiler, the output profiles did not contain new fields such as columnLabel or xstructure. I attached the csv file I used to test. In this dataset, dna_sample_timestamp should be labeled as datetime and location should be labeled as geo-coordinate.

2ivx-z93u.csv

luthfibalaka commented 7 months ago

Is there a way to extend Xsystem to recognize more format for each type?

For instance, columns in the format format {'type': 'Point', 'coordinates': [a, b]} are not recognized as locations, whereas (a, b) is. Or, do we need to create regex patterns for that?

snowgy commented 7 months ago

yes. you can extend Xsystem to support this.

In ddprofiler/app/src/test/java/ddprofiler/analysis/modules/LabelAnalyzerTest.java#L4, there is a path to the regex reference file. Adding more regex expressions to that file should work