google / fhir-data-pipes

A collection of tools for extracting FHIR resources and analytics services on top of that data.
https://google.github.io/fhir-data-pipes/
Apache License 2.0
151 stars 84 forks source link

Changes to support the Windows OS platform #868

Closed chandrashekar-s closed 10 months ago

chandrashekar-s commented 11 months ago

Description of what I changed

Fixes #843

In Windows platform usage of ResourceId.resolve(..) method to resolve paths with glob expressions (containing multiple special characters like **, */* etc) is leading to the error java.nio.file.InvalidPathException: Illegal char <:> at index 2: /C:/.... This is because it internally uses Paths.get() which fails in Windows OS platform to resolve the paths correctly. Refer https://bugs.openjdk.org/browse/JDK-8197918 for details. Also, the documentation of this api ResourceId.resolve(..) does not talk about the support for glob expressions in general. Its best to avoid using this api for resolving paths containing multiple special characters.

As a fix, Instead of creating ResourceId to match for files, we can create string path specs with glob expressions and use this api FileSystems.match(..) to match the files, which documents about the support for glob expressions as well.

Unsolved: There are 2 warnings which are being reported when the pipelines are being executed in Windows platform, the details are documented here #869.

E2E test

Tested the Full run and Incremental Runs in Windows platform. Tested the Full and Incremental runs for GCS File System on Unix platform.

TESTED:

Regression tests on Unix platform for Full and Incremental runs.

Checklist: I completed these to help reviewers :)

chandrashekar-s commented 10 months ago

Thanks @bashir2 for the review. I have updated the code and also responded to your comments. Please have a look.