osmlab / atlas-checks

OSM data integrity checks with Atlas
BSD 3-Clause "New" or "Revised" License
57 stars 83 forks source link

[Bug] Atlas date parsed incorrectly in Construction check causing crash. #685

Closed atiannicelli closed 2 years ago

atiannicelli commented 2 years ago

It looks like the Construction check is not fetching the Atlas Date correctly. The data that this validation is running on has a name "osmbase_2022-08-20". I am not sure why it is parsing this to find the date.

I am seeing this all over the place:

22/08/22 15:48:19 ERROR BaseCheck: ConstructionCheck failed on feature 508294744000000 (508294744).
java.time.format.DateTimeParseException: Text 'osmbase_2022' could not be parsed at index 0
    at java.base/java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:2046)
    at java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1948)
    at java.base/java.time.LocalDate.parse(LocalDate.java:428)
    at org.openstreetmap.atlas.checks.validation.tag.ConstructionCheck.flag(ConstructionCheck.java:140)
    at org.openstreetmap.atlas.checks.base.BaseCheck.check(BaseCheck.java:130)
    at org.openstreetmap.atlas.checks.distributed.RunnableCheck.lambda$run$0(RunnableCheck.java:77)
    at java.base/java.lang.Iterable.forEach(Iterable.java:75)
    at org.openstreetmap.atlas.checks.distributed.RunnableCheck.run(RunnableCheck.java:75)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

It looks to be fetching the name of the atlas instead of the date?? I believe that the value of atlasDateString.get() is "osmbase_2022-08-20"

    // Get the date the atlas was generated, or use today's date as a fallback
    final Optional<String> atlasDateString = object.getAtlas().metaData().getDataVersion();
    final LocalDate comparisonDate = atlasDateString.isPresent()
            && !atlasDateString.get().equals("unknown")
                    ? LocalDate.parse(atlasDateString.get().split("-")[0], ATLAS_DATE_FORMATTER)
                    : TODAYS_DATE;

To reproduce run the ConstructionCheck on any area: (Example EMR command we use)

spark-submit --deploy-mode cluster --master yarn --class org.openstreetmap.atlas.checks.distributed.ShardedIntegrityChecksSparkJob s3://cosmos-atlas-dev/Atlas_Checks/utils/jar/atlas-checks-latest-SNAPSHOT-shaded.jar -input=s3://cosmos-atlas-dev/Atlas_Generation/prod/osmbase_2022-08-20/atlas -output=s3://cosmos-atlas-dev/Atlas_Checks/prod/osmbase_2022-08-20 -outputFormats=flags -countries=ZWE -configFiles=https://raw.githubusercontent.com/atiannicelli/atlas-checks/cosmos/config/configuration.json
vladlemberg commented 2 years ago

@Bentleysb, please assign this task to me.

vladlemberg commented 2 years ago

merged to dev. please close.