search

MobilityData / gtfs-validator

Canonical GTFS Validator project for schedule (static) files.
https://gtfs-validator.mobilitydata.org/
Apache License 2.0
290 stars 101 forks source link

Output fails when output file is open #112

Closed barbeau closed 4 years ago

barbeau commented 4 years ago

Describe the bug If after running the validator you open the output results.json file and then run validator again, the output fails with an error message:

[ERROR] 2020-04-15 17:06:38.743 [main] Main - An exception occurred: C:\git-projects\gtfs-validator\application\cli-app\build\libs\output\results.json

To Reproduce Run from command-line or IntelliJ

  1. Run the validator (java -jar gtfs-validator-v1.1.0-SNAPSHOT.jar -u https://github.com/CUTR-at-USF/bullrunner-gtfs-realtime-generator/raw/master/bullrunner-gtfs.zip -z input.zip -i input -o output)
  2. Open the output/results.json file in an editor like Notepad++
  3. Run the validator again (java -jar gtfs-validator-v1.1.0-SNAPSHOT.jar -u https://github.com/CUTR-at-USF/bullrunner-gtfs-realtime-generator/raw/master/bullrunner-gtfs.zip -z input.zip -i input -o output)

Expected behavior Validation should succeed and overwrite the output file, or the original output file should not be overwritten

Witnessed behavior

The validator deletes the old output file and directory, but fails to output a new output file.

Here's the console output:

Notice{filename='calendar_dates.txt', noticeId='E015', title='Missing required value', description='Missing value for field:service_id marked as required in entity with id:null'}, Notice{filename='calendar_dates.txt', noticeId='E015', title='Missing required value', description='Missing value for field:date marked as required in entity with id:null'}, Notice{filename='calendar_dates.txt', noticeId='E015', title='Missing required value', description='Missing value for field:exception_type marked as required in entity with id:null'}] [ERROR] 2020-04-15 17:06:38.743 [main] Main - An exception occurred: C:\git-projects\gtfs-validator\application\cli-app\build\libs\output\results.json [INFO ] 2020-04-15 17:06:38.750 [main] Main - Took 2038ms PS C:\git-projects\gtfs-validator\application\cli-app\build\libs>

Environment used

barbeau commented 4 years ago

Looking at CleanOrCreatePath.execute(), I don't think that exceptions related to not being able to delete files or create a directory should be stored in the validation results repo - they should be output to the logger/command-line.

barbeau commented 4 years ago

It looks like the error message is only being partially output too here - it should be:

[ERROR] 2020-04-15 17:28:34.059 [main] Main - An exception occurred: java.nio.file.NoSuchFileException: C:\git-projects\gtfs-validator\output\results.json

I just opened https://github.com/MobilityData/gtfs-validator/pull/113 to fix this.

And here's the full stack trace:

java.nio.file.NoSuchFileException: C:\git-projects\gtfs-validator\output\results.json
    at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:85)
    at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
    at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
    at java.base/sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:235)
    at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:478)
    at java.base/java.nio.file.Files.newOutputStream(Files.java:224)
    at org.mobilitydata.gtfsvalidator.db.InMemoryValidationResultRepository.getExporter(InMemoryValidationResultRepository.java:119)
    at org.mobilitydata.gtfsvalidator.usecase.ExportResultAsFile.execute(ExportResultAsFile.java:39)
    at org.mobilitydata.gtfsvalidator.Main.main(Main.java:146)
irees commented 4 years ago

Is this a Windows issue? IIRC Windows does not allow open files to be deleted without a specific open flag.

barbeau commented 4 years ago

@irees Maybe - when testing a fix I ran across these:

...which indicate that immediately after delete sometimes Windows may block Files.createDirectory() with a java.nio.file.AccessDeniedException.

But I'm not sure why this is only happening when I have the file open in an editor and why it doesn't always happen, especially because the file gets deleted anyway even if it's open in editor.

I'm curious if someone can reproduce this on Linux or Mac.

You don't need a release build for this - you can test running from IntelliJ.

ghost commented 4 years ago

On my mac machine I tested this situation by having the file opened in IntelliJ, the content is updated without any issue. with TextEdit the file had to be reopened. In both cases the validator executed without error.

barbeau commented 4 years ago

@fabrice-v You might want to try an external editor - having IntelliJ running the application and having the file open may be a special case.

ghost commented 4 years ago

Yep, as mentionned I also tried with TextEdit and even though the file content didn't update, the validator execution went normaly.