al2na
commented
3 years ago this is a crazy bug : ) !! Do you know any alternatives? like delayedMatrix to move away from tabix, last time I checked they didn't have this overlap functionality we use with tabix.
On Tue, Jan 26, 2021 at 4:22 PM Alexander Blume (Gosdschan) < notifications@github.com> wrote:
It seems as if the caching of uncompressed files introduced with version 1.13.2 causes some problems if the user works with tabix files for more than one CpG context and wants to convert those into memory objects. See here: https://groups.google.com/g/methylkit_discussion/c/UruFjvX89B4/m/_aMsqBC-DwAJ
The problem is caused by the fread.gzipped() function and arises when any two tabix files with the same basename are used in the same session. Once one tabix file is read with fread.gzipped, it will be uncompressed and stored in a (session specific) temporary location for the first time, but subsequent calls to fread.gzipped will reuse the cached uncompressed file. If now, another tabix file with the same basename is supposed to be uncompressed, the cached file with the same basename will be read. Unfortunately this happens unnoticed, as missing rows will be filled with NA's and might cause unnoticed issues downstream.
One (hopefully) simple idea to mitigate this would be to calculate hashes for the compressed file that could become part of the name of the cached files. However we need to make sure to ignore the tabix files header if present.
- calculate hashes for the compressed file
- ignore present tabix files header
- make hash part of cached files name
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/al2na/methylKit/issues/222, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAE32EPWI5WXA73DZPOLORTS33M3TANCNFSM4WTT7UBQ .
alexg9010
It seems as if the caching of uncompressed files introduced with version 1.13.2 causes some problems if the user works with tabix files for more than one CpG context and wants to convert those into memory objects. See here: https://groups.google.com/g/methylkit_discussion/c/UruFjvX89B4/m/_aMsqBC-DwAJ
The problem is caused by the
fread.gzipped()function and arises when any two tabix files with the same basename are used in the same session. Once one tabix file is read with fread.gzipped, it will be uncompressed and stored in a (session specific) temporary location for the first time, but subsequent calls to fread.gzipped will reuse the cached uncompressed file. If now, another tabix file with the same basename is supposed to be uncompressed, the cached file with the same basename will be read. Unfortunately this happens unnoticed, as missing rows will be filled with NA's and might cause unnoticed issues downstream.One (hopefully) simple idea to mitigate this would be to calculate hashes for the compressed file that could become part of the name of the cached files. However we need to make sure to ignore the tabix files header if present.