Tcorr Issue

[nwf9]

Hi Matias,

I have an issue with tcorr. You will find below the error messages.

python AppCompatProcessor.py shimcachedb tcorr "net1.exe" 2018-07-18 16:56:55,567 INFO -------------------------------Log started------------------------------- 2018-07-18 16:56:55,569 INFO Starting to process request... 2018-07-18 16:56:56,492 INFO Searching for AppCompat temporal correlations on FileName: net1.exe => [875 hits] net1.exe: [#####--------------------] 21.43% Traceback (most recent call last): File "AppCompatProcessor.py", line 1400, in 2018-07-18 16:56:57,113 ERROR SQLITE error: disk I/O error [INSERT INTO TemporalCollateral VALUES (NULL,426537, 0, 0, 0, 0)] Traceback (most recent call last): File "/mnt/c/xxx/appcompatprocessor-master/appDB.py", line 408, in Execute self.connRAW.execute(query) OperationalError: disk I/O error main(sys.argv[1:]) File "AppCompatProcessor.py", line 1341, in main ret = appTcorr(options.tcorr_filename.decode(sys.getfilesystemencoding()), options.sqlTweak, DB, True, options.window) File "AppCompatProcessor.py", line 413, in appTcorr appCompatCorrelationData = appTcorrAppCompat(fileName, sqlTweak, DB, directCorrelation, reconWindow) File "AppCompatProcessor.py", line 468, in appTcorrAppCompat PopulateAppCompatTemporalCollaterals(fileName, sqlTweak, DB, collateralDBTableName, tcorrWindow) File "AppCompatProcessor.py", line 544, in PopulateAppCompatTemporalCollaterals DB.Execute("INSERT INTO " + collateralDBTableName + " VALUES (NULL,%s, 0, 0, 0, 0)" % (rowID)) File "/mnt/c/xxx/appDB.py", line 408, in Execute self.connRAW.execute(query) sqlite3.OperationalError: disk I/O error

[mbevilacqua]

That's weird. Could you report OS and confirm the mount point is RW? Would be good to try to copy the database to /tmp and try there again to exclude mount issues.

[nwf9]

Yes the mountpoint is RW and i have Ubuntu 4.4.0 kernel. I have the same issue when i move the db to /tmp like below. They are no issue with ptcorr.

python AppCompatProcessor.py /tmp/shimcachedb tcorr "svchost.exe" 2018-07-19 11:14:41,830 INFO -------------------------------Log started------------------------------- 2018-07-19 11:14:41,831 INFO Starting to process request... 2018-07-19 11:14:42,842 INFO Searching for AppCompat temporal correlations on FileName: svchost.exe => [2211 hits] Traceback (most recent call last): File "AppCompatProcessor.py", line 1400, in 2018-07-19 11:15:03,669 ERROR SQLITE error: table TemporalInverseCollateral already exists [CREATE TABLE TemporalInverseCollateral (TempID INTEGER PRIMARY KEY, RowID integer, Before integer, After integer, Weight integer, InvBond integer, FOREIGN KEY(RowID) REFERENCES Entries(RowID))] Traceback (most recent call last): File "/home/tools/appcompatprocessor-master/appDB.py", line 408, in Execute self.connRAW.execute(query) OperationalError: table TemporalInverseCollateral already exists main(sys.argv[1:]) File "AppCompatProcessor.py", line 1341, in main ret = appTcorr(options.tcorr_filename.decode(sys.getfilesystemencoding()), options.sqlTweak, DB, True, options.window) File "AppCompatProcessor.py", line 413, in appTcorr appCompatCorrelationData = appTcorrAppCompat(fileName, sqlTweak, DB, directCorrelation, reconWindow) File "AppCompatProcessor.py", line 489, in appTcorrAppCompat inverseCorrelationData = appTcorrAppCompat(directCorrelationData[i][3], sqlTweak, DB, False, tcorrWindow) File "AppCompatProcessor.py", line 460, in appTcorrAppCompat FOREIGN KEY(RowID) REFERENCES Entries(RowID))''') File "/home/tools/appcompatprocessor-master/appDB.py", line 408, in Execute self.connRAW.execute(query) sqlite3.OperationalError: table TemporalInverseCollateral already exists

[TridentFactor]

That happens whenever I interrupt Appcompat from completing a Tcorr task. Re ingest your data in a new DB and try not to interrupt any Tcorr task till they are finished, The issue will never happen again.

[mbevilacqua]

oh, well yeah interrupting APC is a really bad think to do :) While I try to detect being killed and exist of multiprocessing processes gracefully it's non-trivial to do and that part of the code if not something I trust totally. The most challenging piece is that multiprocessing does not behave exactly in the same way under different OS's. Sort of sticking to OSX for the time being tbh. Planning to formally add unittests on dockerized Linux containers sometime this year though as we're pushing in that direction. Closing for now.