search

ohdsi-studies / PioneerWatchfulWaiting

This study is part of the joint PIONEER - EHDEN - OHDSI studyathon in March 2021, and aims to advance understanding of clinical management and outcomes of watchful waiting in prostate cancer.
Apache License 2.0
7 stars 18 forks source link

Error: cannot open file for writing: * '...\CreatedDiagnostics.csv' #75

Closed peterprinsen-iknl closed 2 years ago

peterprinsen-iknl commented 3 years ago

I get an error in runCohortDiagnostics: Cannot open file for writing: * 'C:\Data\PIONEER\studyathon\output\NCR\RecordKeeping\CreatedDiagnostics.csv' (see attached log file).

I cannot really figure out what is going on. The file exists so that shouldn't be the problem. The bundled results file is created but it doesn't seem that all csv files are there (in the zip file but also in the results folder). I haven't had this problem before but I am running the code on a much larger data set now (all cancer patients in the NCR instead of only the prostate cancer patients).

I am running version 0.4.2.2 of the package, so not the newest one. I could update to the newest one but running it takes a while and it doesn't seem to be a problem that has occurred before so I don't expect it to be solved in the newer version.

cohortDiagnosticsLog.txt

peterprinsen-iknl commented 3 years ago

I get the same error with version 0.4.3. My suspicion is that it has something to do with the fact that we don't have visits in our OMOP-CDM because it happens in getVisitContect.

keesvanbochove commented 3 years ago

Hello @peterprinsen-iknl, this is a bit curious indeed - could it be that you had this file open in an editor and therefore the file system wouldn't let R (re)create the file? See https://docs.microsoft.com/en-us/troubleshoot/windows-server/backup-and-storage/cannot-delete-file-folder-on-ntfs-file-system for possible reasons - some edge cases would include if a folder exists with that exact name. You can also try to reproduce this in R by trying to write to the file using cat or writeLines - note that you would have to escape the slashes. From the log, it seems that writing to any of the other output files works.

peterprinsen-iknl commented 3 years ago

I am getting the error earlier on in the script now, with the file InstantiatedCohorts.csv. Tim Hulsen helped me debug and it seems to be an issue with readr::read_csv in Incremental.R. If you read the documentation on https://readr.tidyverse.org/reference/read_delim.html it states: "Note, if you later write to the same file you read from you need to set lazy = FALSE. On Windows the file will be locked and on other systems the memory map will become invalid".

keesvanbochove commented 2 years ago

@peterprinsen-iknl this should be fixed in the newest version, because Incremental.R is changed to use fread instead of read.csv. Could you try re-running with this newest version? It should also run a bit faster.

peterprinsen-iknl commented 2 years ago

@keesvanbochove this seems to work. No errors on the small data set. I will now run it on the full data set.