SanthoshBala
commented
7 years ago Note for others: will need to integrate this in to the deploy workflow
Closed SanthoshBala closed 6 years ago
SanthoshBala
commented
7 years ago Note for others: will need to integrate this in to the deploy workflow
SanthoshBala
commented
7 years ago Will need to triage the failing tests with @jonc101
SanthoshBala
commented
7 years ago Should probably make sure we can run this against both medinfo and scripts
SanthoshBala
commented
7 years ago Notes: • medinfo.common.test.TestProcessManager is empty – can we remove it? • TestDecayingWindows fails because ResetModel no longer exists?? •
jonc101
commented
7 years ago I think I merged the ResetModel functionality into DataManager.py. Can the test case call just be redirected to that version?
TestProcessManager can be removed or commented out for now. Was a placeholder to fill in some parallel process management functionality, but the prior dev never got around to working on it.
Jonathan H. Chen, MD, PhD Stanford Department of Medicine http://web.stanford.edu/~jonc101
From: Santhosh Balasubramanian Sent: Thursday, October 12, 2017 12:00 PM To: HealthRex/CDSS Cc: jonc101; Mention Subject: Re: [HealthRex/CDSS] Write script to do mass regression test, forrunning pre-commit (#2)
Notes: • medinfo.common.test.TestProcessManager is empty – can we remove it? • TestDecayingWindows fails because ResetModel no longer exists?? • — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.
SanthoshBala
commented
7 years ago Yeah, I'll do both of those fixes.
On a naive run of the full test suite, 6 fail and 25 throw errors out of 109 tests. Some of those are quick fixes (e.g. I hadn't initialized the tables in CpoeSimulation.sql, and didn't have some libraries installed), but others seem a bit more involved (e.g. some of the tests seem to not clean up their records, so we end up trying to insert duplicate records).
As I'm going through, trying to move as much as possible into a setup.sh script that should take care of all the library dependencies, DB initialization, etc. for a fully functional dev environment.
SanthoshBala
commented
7 years ago Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/cpoe/analysis/test/TestTopicModelAnalysis.py", line 164, in test_topicModelAnalysis self.analyzer.main(argv); File "/Users/sbala/code/CDSS/medinfo/cpoe/analysis/TopicModelAnalysis.py", line 171, in main formatter.formatResultDicts( analysisResults, colNames ); File "/Users/sbala/code/CDSS/medinfo/db/ResultsFormatter.py", line 60, in formatResultDicts for self.currRow, rowDict in enumerate(resultDicts): File "/Users/sbala/code/CDSS/medinfo/cpoe/analysis/TopicModelAnalysis.py", line 63, in call preparer File "/Users/sbala/code/CDSS/medinfo/cpoe/analysis/TopicModelAnalysis.py", line 93, in analyzePatientItems recommendedData = recommender( recQuery ); File "/Users/sbala/code/CDSS/medinfo/cpoe/TopicModelRecommender.py", line 69, in call self.weightByItemIdByTopicId = self.modeler.generateWeightByItemIdByTopicId(self.model, query.itemsPerCluster); File "/Users/sbala/code/CDSS/medinfo/cpoe/TopicModel.py", line 140, in generateWeightByItemIdByTopicId for (topicId, topicItems) in self.enumerateTopics(model, itemsPerCluster): File "/Users/sbala/code/CDSS/medinfo/cpoe/TopicModel.py", line 128, in enumerateTopics for (itemWeight, itemId) in flipTopicItems: TypeError: 'int' object is not iterable
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/cpoe/analysis/test/TestTopicModelAnalysis.py", line 232, in test_topicModelAnalysis_numRecsByOrderSet self.analyzer.main(argv); File "/Users/sbala/code/CDSS/medinfo/cpoe/analysis/TopicModelAnalysis.py", line 171, in main formatter.formatResultDicts( analysisResults, colNames ); File "/Users/sbala/code/CDSS/medinfo/db/ResultsFormatter.py", line 60, in formatResultDicts for self.currRow, rowDict in enumerate(resultDicts): File "/Users/sbala/code/CDSS/medinfo/cpoe/analysis/TopicModelAnalysis.py", line 63, in call preparer File "/Users/sbala/code/CDSS/medinfo/cpoe/analysis/TopicModelAnalysis.py", line 93, in analyzePatientItems recommendedData = recommender( recQuery ); File "/Users/sbala/code/CDSS/medinfo/cpoe/TopicModelRecommender.py", line 69, in call self.weightByItemIdByTopicId = self.modeler.generateWeightByItemIdByTopicId(self.model, query.itemsPerCluster); File "/Users/sbala/code/CDSS/medinfo/cpoe/TopicModel.py", line 140, in generateWeightByItemIdByTopicId for (topicId, topicItems) in self.enumerateTopics(model, itemsPerCluster): File "/Users/sbala/code/CDSS/medinfo/cpoe/TopicModel.py", line 128, in enumerateTopics for (itemWeight, itemId) in flipTopicItems: TypeError: 'int' object is not iterable
ImportError: Failed to import test module: medinfo.cpoe.test.TestDecayingWindows
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/loader.py", line 254, in _find_tests
module = self._get_module_from_name(name)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
import(name)
File "/Users/sbala/code/CDSS/medinfo/cpoe/test/TestDecayingWindows.py", line 19, in
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/cpoe/test/TestSimManager.py", line 26, in setUp self.purgeTestRecords(); File "/Users/sbala/code/CDSS/medinfo/cpoe/test/TestSimManager.py", line 247, in purgeTestRecords DBUtil.execute("delete from sim_note where sim_note_id < 0"); File "/Users/sbala/code/CDSS/medinfo/db/DBUtil.py", line 261, in execute cur.execute( query, parameters ) ProgrammingError: relation "sim_note" does not exist LINE 1: delete from sim_note where sim_note_id < 0 ^
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/cpoe/test/TestSimManager.py", line 26, in setUp self.purgeTestRecords(); File "/Users/sbala/code/CDSS/medinfo/cpoe/test/TestSimManager.py", line 247, in purgeTestRecords DBUtil.execute("delete from sim_note where sim_note_id < 0"); File "/Users/sbala/code/CDSS/medinfo/db/DBUtil.py", line 261, in execute cur.execute( query, parameters ) ProgrammingError: relation "sim_note" does not exist LINE 1: delete from sim_note where sim_note_id < 0 ^
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/cpoe/test/TestSimManager.py", line 26, in setUp self.purgeTestRecords(); File "/Users/sbala/code/CDSS/medinfo/cpoe/test/TestSimManager.py", line 247, in purgeTestRecords DBUtil.execute("delete from sim_note where sim_note_id < 0"); File "/Users/sbala/code/CDSS/medinfo/db/DBUtil.py", line 261, in execute cur.execute( query, parameters ) ProgrammingError: relation "sim_note" does not exist LINE 1: delete from sim_note where sim_note_id < 0 ^
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/cpoe/test/TestTopicModel.py", line 98, in test_topicModel self.instance.main(subargv); File "/Users/sbala/code/CDSS/medinfo/cpoe/TopicModel.py", line 260, in main self.printTopicsToFile(model, docCountByWordId, topicFile, itemsPerCluster); File "/Users/sbala/code/CDSS/medinfo/cpoe/TopicModel.py", line 151, in printTopicsToFile for (topicId, topicItems) in self.enumerateTopics(model, itemsPerCluster): File "/Users/sbala/code/CDSS/medinfo/cpoe/TopicModel.py", line 128, in enumerateTopics for (itemWeight, itemId) in flipTopicItems: TypeError: 'int' object is not iterable
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/cpoe/test/TestTripleAssociationAnalysis.py", line 121, in test_analyzePatientItems self.analyzer.analyzePatientItems( [-11111], (-15,-14), -16 ); # Count associations that result in given sequence of items File "/Users/sbala/code/CDSS/medinfo/cpoe/TripleAssociationAnalysis.py", line 64, in analyzePatientItems self.updateItemAssociationsBuffer(itemIdSequence, virtualItemId, patientItemList, updateBuffer, linkedItemIdsByBaseId, progress=progress); File "/Users/sbala/code/CDSS/medinfo/cpoe/TripleAssociationAnalysis.py", line 115, in updateItemAssociationsBuffer self.updateClinicalItemAssociationBuffer( patientItem1, patientItem2, isNewSubsequentItem, isNewPair, isNewPairWithinEncounter, updateBuffer, itemIdPair=itemIdPair ); TypeError: updateClinicalItemAssociationBuffer() takes at least 8 arguments (8 given)
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/dataconversion/test/TestDataExtractor_tempTable.py", line 172, in setUp self.extractor = DataExtractor(self.outputFileName); # Instance to test on TypeError: init() takes exactly 1 argument (2 given)
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/dataconversion/test/TestDataExtractor_tempTable.py", line 75, in setUp DBUtil.insertFile( StringIO(dataTextStr), "clinical_item_category", delim="\t", columnNames=columns); File "/Users/sbala/code/CDSS/medinfo/db/DBUtil.py", line 475, in insertFile cur.execute(sql,tuple(params)) IntegrityError: duplicate key value violates unique constraint "clinical_item_category_pkey" DETAIL: Key (clinical_item_category_id)=(-100) already exists.
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/dataconversion/test/TestDataExtractor_tempTable.py", line 75, in setUp DBUtil.insertFile( StringIO(dataTextStr), "clinical_item_category", delim="\t", columnNames=columns); File "/Users/sbala/code/CDSS/medinfo/db/DBUtil.py", line 475, in insertFile cur.execute(sql,tuple(params)) IntegrityError: duplicate key value violates unique constraint "clinical_item_category_pkey" DETAIL: Key (clinical_item_category_id)=(-100) already exists.
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/dataconversion/test/TestDataExtractor_tempTable.py", line 75, in setUp DBUtil.insertFile( StringIO(dataTextStr), "clinical_item_category", delim="\t", columnNames=columns); File "/Users/sbala/code/CDSS/medinfo/db/DBUtil.py", line 475, in insertFile cur.execute(sql,tuple(params)) IntegrityError: duplicate key value violates unique constraint "clinical_item_category_pkey" DETAIL: Key (clinical_item_category_id)=(-100) already exists.
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/dataconversion/test/TestDataExtractor_tempTable.py", line 75, in setUp DBUtil.insertFile( StringIO(dataTextStr), "clinical_item_category", delim="\t", columnNames=columns); File "/Users/sbala/code/CDSS/medinfo/db/DBUtil.py", line 475, in insertFile cur.execute(sql,tuple(params)) IntegrityError: duplicate key value violates unique constraint "clinical_item_category_pkey" DETAIL: Key (clinical_item_category_id)=(-100) already exists.
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/dataconversion/test/TestDataExtractor_tempTable.py", line 75, in setUp DBUtil.insertFile( StringIO(dataTextStr), "clinical_item_category", delim="\t", columnNames=columns); File "/Users/sbala/code/CDSS/medinfo/db/DBUtil.py", line 475, in insertFile cur.execute(sql,tuple(params)) IntegrityError: duplicate key value violates unique constraint "clinical_item_category_pkey" DETAIL: Key (clinical_item_category_id)=(-100) already exists.
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/dataconversion/test/TestFinding.py", line 279, in test_YTTFinding query = YTTFinding(sampleFindingModel).writeSubquery(findingIndex) TypeError: writeSubquery() takes exactly 1 argument (2 given)
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/dataconversion/test/TestPreliminaryQuery.py", line 74, in test_clinicalPreliminaryQueryByCategory instance = ClinicalPreliminaryQueryByCategory(self.patientIds, clinicalItemCategoryIds) NameError: global name 'ClinicalPreliminaryQueryByCategory' is not defined
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/dataconversion/test/TestPreliminaryQuery.py", line 49, in test_clinicalPreliminaryQueryByName instance = ClinicalPreliminaryQueryByName(self.patientIds, clinicalItemNames) NameError: global name 'ClinicalPreliminaryQueryByName' is not defined
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/dataconversion/test/TestPreliminaryQuery.py", line 125, in test_flowsheetPreliminaryQuery instance = FlowsheetPreliminaryQuery(self.patientIds, flowsheetNames) NameError: global name 'FlowsheetPreliminaryQuery' is not defined
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/dataconversion/test/TestPreliminaryQuery.py", line 152, in test_ivFluidPreliminaryQuery instance = IVFluidPreliminaryQuery([-123], medicationGroups) NameError: global name 'IVFluidPreliminaryQuery' is not defined
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/dataconversion/test/TestPreliminaryQuery.py", line 100, in test_labPreliminaryQuery instance = LabPreliminaryQuery(self.patientIds, baseNames) NameError: global name 'LabPreliminaryQuery' is not defined
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/dataconversion/test/TestSTRIDEAccessLogConversion.py", line 41, in setUp """ % TEST_SOURCE_TABLE File "/Users/sbala/code/CDSS/medinfo/db/DBUtil.py", line 261, in execute cur.execute( query, parameters ) ProgrammingError: type "datetime" does not exist LINE 6: ACCESS_DATETIME datetime, ^
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/dataconversion/test/TestSTRIDEDxListConversion.py", line 72, in setUp (dataItemId, isNew) = DBUtil.findOrInsertItem("stride_icd9_cm", dataModel, retrieveCol="cui" ); File "/Users/sbala/code/CDSS/medinfo/db/DBUtil.py", line 682, in findOrInsertItem cur.execute( searchQuery, searchParams ); ProgrammingError: relation "stride_icd9_cm" does not exist LINE 2: FROM stride_icd9_cm ^
ImportError: Failed to import test module: medinfo.dataconversion.test.TestVAReminderToQuery
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/loader.py", line 254, in _find_tests
module = self._get_module_from_name(name)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
import(name)
File "/Users/sbala/code/CDSS/medinfo/dataconversion/test/TestVAReminderToQuery.py", line 17, in
ImportError: Failed to import test module: medinfo.db.test.TestDBUtil
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/loader.py", line 254, in _find_tests
module = self._get_module_from_name(name)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
import(name)
File "/Users/sbala/code/CDSS/medinfo/db/test/TestDBUtil.py", line 10, in
ImportError: Failed to import test module: medinfo.db.test.TestResultsFormatter
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/loader.py", line 254, in _find_tests
module = self._get_module_from_name(name)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
import(name)
File "/Users/sbala/code/CDSS/medinfo/db/test/TestResultsFormatter.py", line 10, in
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/dataconversion/test/TestFinding.py", line 246, in test_LABFinding self.assertEqualFile(StringIO(positiveExpectedOutput), StringIO(queryText), whitespace=False) File "/Users/sbala/code/CDSS/medinfo/common/test/Util.py", line 117, in assertEqualFile self.assertEqualNonWhitespace( verifyLine, sampleLine ); File "/Users/sbala/code/CDSS/medinfo/common/test/Util.py", line 130, in assertEqualNonWhitespace self.assertEqual(verifySimpleStr, sampleSimpleStr, errorMsg); AssertionError: FROM [ORD_Chan_201406081D].[Src].[Chem_PatientLabChem] AS t1
!= FROM [ORD_Chan_201406081D].[Src].[Chem_PatientLabChem] as t1
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/dataconversion/test/TestFinding.py", line 424, in test_PSDRUGFinding self.assertEqualFile(StringIO(positiveExpectedOutput), StringIO(queryText), whitespace=False) File "/Users/sbala/code/CDSS/medinfo/common/test/Util.py", line 117, in assertEqualFile self.assertEqualNonWhitespace( verifyLine, sampleLine ); File "/Users/sbala/code/CDSS/medinfo/common/test/Util.py", line 130, in assertEqualNonWhitespace self.assertEqual(verifySimpleStr, sampleSimpleStr, errorMsg); AssertionError: SELECT t1.Sta3n, t1.RxOutpatFillSID, t1.FillDateTime AS DateTime, t1.PatientSID, t2.LocalDrugIEN AS FindingIEN
!= SELECT t1.Sta3n, t1.FillDateTime AS DateTime, t1.PatientSID, t2.LocalDrugIEN AS FindingIEN
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/dataconversion/test/TestFinding.py", line 210, in test_PSNDFFinding self.assertEqualFile(StringIO(positiveExpectedOutput), StringIO(queryText), whitespace=False) File "/Users/sbala/code/CDSS/medinfo/common/test/Util.py", line 117, in assertEqualFile self.assertEqualNonWhitespace( verifyLine, sampleLine ); File "/Users/sbala/code/CDSS/medinfo/common/test/Util.py", line 130, in assertEqualNonWhitespace self.assertEqual(verifySimpleStr, sampleSimpleStr, errorMsg); AssertionError: FROM [ORD_Chan_201406081D].[Src].[RxOut_RxOutpatFill] AS t1
!= FROM [ORD_Chan_201406081D].[Src].[RxOut_RxOutpatFill] as t1
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/dataconversion/test/TestFinding.py", line 169, in test_PXDFinding self.assertEqualFile(StringIO(positiveExpectedOutput), StringIO(queryText), whitespace=False) File "/Users/sbala/code/CDSS/medinfo/common/test/Util.py", line 117, in assertEqualFile self.assertEqualNonWhitespace( verifyLine, sampleLine ); File "/Users/sbala/code/CDSS/medinfo/common/test/Util.py", line 130, in assertEqualNonWhitespace self.assertEqual(verifySimpleStr, sampleSimpleStr, errorMsg); AssertionError: SELECT t1.Sta3n, t1.VDiagnosisDateTime AS DateTime, t1.PatientSID
!= SELECT t1.Sta3n, t1.VDiagnosisDateTime AS DateTime, t1.PatientSID, '40' AS FindingIEN
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/dataconversion/test/TestFinding.py", line 69, in test_PXRMDFinding self.assertEqualFile(StringIO(positiveExpectedOutput), StringIO(queryText), whitespace=False) File "/Users/sbala/code/CDSS/medinfo/common/test/Util.py", line 117, in assertEqualFile self.assertEqualNonWhitespace( verifyLine, sampleLine ); File "/Users/sbala/code/CDSS/medinfo/common/test/Util.py", line 130, in assertEqualNonWhitespace self.assertEqual(verifySimpleStr, sampleSimpleStr, errorMsg); AssertionError: FROM [ORD_Chan_201406081D].[Src].[Chem_PatientLabChem] AS t1
!= FROM [ORD_Chan_201406081D].[Src].[Chem_PatientLabChem] as t1
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/dataconversion/test/TestSTRIDEOrderMedConversion.py", line 455, in test_dataConversion_denormalized self.assertEqualTable( expectedData, actualData ); File "/Users/sbala/code/CDSS/medinfo/common/test/Util.py", line 172, in assertEqualTable self.assertEqual(verifyItem, sampleItem) AssertionError: -888699365 != -2919978438137248229L
Ran 109 tests in 35.344s
FAILED (failures=6, errors=25) `
jonc101
commented
7 years ago I would not try to "fix" all of these test failures. Some represent obsolete or separate code that isn't being used anymore anyway. Notes below on ones to skip:
ERROR: test_labPreliminaryQuery (medinfo.dataconversion.test.TestPreliminaryQuery.TestPreliminaryQuery)
These all look like dependencies of the above version. I think these can similarly be removed from the active code base.
ERROR: medinfo.dataconversion.test.TestVAReminderToQuery (unittest.loader.ModuleImportFailure)
The above is related to a separate project the prior dev was working on for a collaboration regarding parsing clinical decision support alerts in VA data. I think these can be removed from our active repository. The people left working on this are still using the old SVN repository.
====================================================================== ERROR: test_dataConversion (medinfo.dataconversion.test.TestSTRIDEAccessLogConversion.TestSTRIDEAccessLogConversion)
Maybe just comment this one out (or make the test suite method return an empty list). This is older code working against a MySQL database (thus the compatibility errors). This may still end up being useful as a reference for future work however. So don't want to remove it completely yet.
SanthoshBala
commented
7 years ago Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/cpoe/analysis/test/TestTopicModelAnalysis.py", line 164, in test_topicModelAnalysis self.analyzer.main(argv); File "/Users/sbala/code/CDSS/medinfo/cpoe/analysis/TopicModelAnalysis.py", line 171, in main formatter.formatResultDicts( analysisResults, colNames ); File "/Users/sbala/code/CDSS/medinfo/db/ResultsFormatter.py", line 60, in formatResultDicts for self.currRow, rowDict in enumerate(resultDicts): File "/Users/sbala/code/CDSS/medinfo/cpoe/analysis/TopicModelAnalysis.py", line 63, in call preparer File "/Users/sbala/code/CDSS/medinfo/cpoe/analysis/TopicModelAnalysis.py", line 93, in analyzePatientItems recommendedData = recommender( recQuery ); File "/Users/sbala/code/CDSS/medinfo/cpoe/TopicModelRecommender.py", line 69, in call self.weightByItemIdByTopicId = self.modeler.generateWeightByItemIdByTopicId(self.model, query.itemsPerCluster); File "/Users/sbala/code/CDSS/medinfo/cpoe/TopicModel.py", line 140, in generateWeightByItemIdByTopicId for (topicId, topicItems) in self.enumerateTopics(model, itemsPerCluster): File "/Users/sbala/code/CDSS/medinfo/cpoe/TopicModel.py", line 128, in enumerateTopics for (itemWeight, itemId) in flipTopicItems: TypeError: 'int' object is not iterable
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/cpoe/analysis/test/TestTopicModelAnalysis.py", line 232, in test_topicModelAnalysis_numRecsByOrderSet self.analyzer.main(argv); File "/Users/sbala/code/CDSS/medinfo/cpoe/analysis/TopicModelAnalysis.py", line 171, in main formatter.formatResultDicts( analysisResults, colNames ); File "/Users/sbala/code/CDSS/medinfo/db/ResultsFormatter.py", line 60, in formatResultDicts for self.currRow, rowDict in enumerate(resultDicts): File "/Users/sbala/code/CDSS/medinfo/cpoe/analysis/TopicModelAnalysis.py", line 63, in call preparer File "/Users/sbala/code/CDSS/medinfo/cpoe/analysis/TopicModelAnalysis.py", line 93, in analyzePatientItems recommendedData = recommender( recQuery ); File "/Users/sbala/code/CDSS/medinfo/cpoe/TopicModelRecommender.py", line 69, in call self.weightByItemIdByTopicId = self.modeler.generateWeightByItemIdByTopicId(self.model, query.itemsPerCluster); File "/Users/sbala/code/CDSS/medinfo/cpoe/TopicModel.py", line 140, in generateWeightByItemIdByTopicId for (topicId, topicItems) in self.enumerateTopics(model, itemsPerCluster): File "/Users/sbala/code/CDSS/medinfo/cpoe/TopicModel.py", line 128, in enumerateTopics for (itemWeight, itemId) in flipTopicItems: TypeError: 'int' object is not iterable
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/cpoe/test/TestDecayingWindows.py", line 143, in test_decayingWindowsFromBuffer self.decayAnalyzer.decayAnalyzePatientItems(decayAnalysisOptions) File "/Users/sbala/code/CDSS/medinfo/cpoe/DecayingWindows.py", line 202, in decayAnalyzePatientItems currentBuffer = instance.mergeBuffers(currentBuffer, bufferOneDelta) File "/Users/sbala/code/CDSS/medinfo/cpoe/AssociationAnalysis.py", line 359, in mergeBuffers if "incrementDataByItemIdPair" not in bufferTwo: TypeError: argument of type 'NoneType' is not iterable
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/cpoe/test/TestSimManager.py", line 229, in setUp DBUtil.insertFile( StringIO(dataTextStr), "sim_state_result", delim=";"); File "/Users/sbala/code/CDSS/medinfo/db/DBUtil.py", line 475, in insertFile cur.execute(sql,tuple(params)) IntegrityError: insert or update on table "sim_state_result" violates foreign key constraint "sim_state_result_state_fkey" DETAIL: Key (sim_state_id)=(0) is not present in table "sim_state".
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/cpoe/test/TestSimManager.py", line 229, in setUp DBUtil.insertFile( StringIO(dataTextStr), "sim_state_result", delim=";"); File "/Users/sbala/code/CDSS/medinfo/db/DBUtil.py", line 475, in insertFile cur.execute(sql,tuple(params)) IntegrityError: insert or update on table "sim_state_result" violates foreign key constraint "sim_state_result_state_fkey" DETAIL: Key (sim_state_id)=(0) is not present in table "sim_state".
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/cpoe/test/TestSimManager.py", line 229, in setUp DBUtil.insertFile( StringIO(dataTextStr), "sim_state_result", delim=";"); File "/Users/sbala/code/CDSS/medinfo/db/DBUtil.py", line 475, in insertFile cur.execute(sql,tuple(params)) IntegrityError: insert or update on table "sim_state_result" violates foreign key constraint "sim_state_result_state_fkey" DETAIL: Key (sim_state_id)=(0) is not present in table "sim_state".
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/cpoe/test/TestTopicModel.py", line 98, in test_topicModel self.instance.main(subargv); File "/Users/sbala/code/CDSS/medinfo/cpoe/TopicModel.py", line 260, in main self.printTopicsToFile(model, docCountByWordId, topicFile, itemsPerCluster); File "/Users/sbala/code/CDSS/medinfo/cpoe/TopicModel.py", line 151, in printTopicsToFile for (topicId, topicItems) in self.enumerateTopics(model, itemsPerCluster): File "/Users/sbala/code/CDSS/medinfo/cpoe/TopicModel.py", line 128, in enumerateTopics for (itemWeight, itemId) in flipTopicItems: TypeError: 'int' object is not iterable
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/cpoe/test/TestTripleAssociationAnalysis.py", line 121, in test_analyzePatientItems self.analyzer.analyzePatientItems( [-11111], (-15,-14), -16 ); # Count associations that result in given sequence of items File "/Users/sbala/code/CDSS/medinfo/cpoe/TripleAssociationAnalysis.py", line 64, in analyzePatientItems self.updateItemAssociationsBuffer(itemIdSequence, virtualItemId, patientItemList, updateBuffer, linkedItemIdsByBaseId, progress=progress); File "/Users/sbala/code/CDSS/medinfo/cpoe/TripleAssociationAnalysis.py", line 115, in updateItemAssociationsBuffer self.updateClinicalItemAssociationBuffer( patientItem1, patientItem2, isNewSubsequentItem, isNewPair, isNewPairWithinEncounter, updateBuffer, itemIdPair=itemIdPair ); TypeError: updateClinicalItemAssociationBuffer() takes at least 8 arguments (8 given)
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/dataconversion/test/TestSTRIDEDxListConversion.py", line 160, in test_dataConversion self.assertEqualTable( expectedData, actualData ); File "/Users/sbala/code/CDSS/medinfo/common/test/Util.py", line 172, in assertEqualTable self.assertEqual(verifyItem, sampleItem) AssertionError: -131014753610 != -131016557370L
Traceback (most recent call last): File "/Users/sbala/code/CDSS/medinfo/dataconversion/test/TestSTRIDEOrderMedConversion.py", line 455, in test_dataConversion_denormalized self.assertEqualTable( expectedData, actualData ); File "/Users/sbala/code/CDSS/medinfo/common/test/Util.py", line 172, in assertEqualTable self.assertEqual(verifyItem, sampleItem) AssertionError: -888699365 != -2919978438137248229L
Ran 102 tests in 39.032s
FAILED (failures=2, errors=8)
SanthoshBala
commented
7 years ago @jonc101 this is down to 2 failures and 8 errors. Some of these might still be due to quirks in my local environment, but anything obvious I should try?
jonc101
commented
6 years ago I'm going to start running through the mass unit test to make sure everything is running, or otherwise deprecated if obsolete. Ran into a couple errors on the FeatureMatrixFactory, looks like issues with temporary file naming not allowing colons from timestamps. Related to that, update the unit test to cleanup any temporary files when done. For example:
fmf.patient_episodes.tsv fmf.patient_list.tsv fmf.patient_PerfItem300.tsv fmf.patient_PerfItem400.tsv fmf.patient_PerfItem500.tsv patient_list.tsv
Traceback (most recent call last): File "C:\HealthRex\CDSS\medinfo\dataconversion\test\TestFeatureMatrixFactory.py", line 271, in test_build_FeatureMatrix_multiLabTest factory.addLabResultFeatures(labBaseNames, False, preTimeDelta, postTimeDelta) File "medinfo\dataconversion\FeatureMatrixFactory.py", line 419, in addLabResultFeatures tempFile = open(tempFileName, "w") IOError: [Errno 22] invalid mode ('w') or filename: 'fmf.patientTNI-CR-LAC-90 days, 0:00:00_0:00:00.tsv'
Traceback (most recent call last): File "C:\HealthRex\CDSS\medinfo\dataconversion\test\TestFeatureMatrixFactory.py", line 389, in test_performance factory.addLabResultFeatures(["Foo"], preTimeDelta, postTimeDelta) File "medinfo\dataconversion\FeatureMatrixFactory.py", line 419, in addLabResultFeatures tempFile = open(tempFileName, "w") IOError: [Errno 22] invalid mode ('w') or filename: 'fmf.patient_Foo_0:00:00_None.tsv'
jonc101
commented
6 years ago I just pushed a bunch of edits to the remaining failing unit tests. Other than the bugs in the FeatureMatrixFactory items above, it looks like everything under the medinfo directory is working now. @SanthoshBala can you patch the FeatureMatrixFactory tests, then we'll finally get back to a stable state with all unit tests passing! :)
SanthoshBala
commented
6 years ago @jonc101 mind pulling and re-running TestCDSS.py?
FeatureMatrixFactory is passing for me, but I'm getting (non-fatal) errors in TestDataManager.test_compositeRelated and TestSTRIDEOrderMedConversion.test_dataConversion_denormalized fails.
Almost there! =)
jonc101
commented
6 years ago Pretty close. Should just be a few minor edits.
TestDataManager.test_compositeRelated It's generating expected error messages. I believe the test is deliberately creating duplicate database entries to assess how the application copes with them.
TestSTRIDEOrderMedConversion.test_dataConversion_denormalized This one passed for me without incident.
TestFeatureMatrixFactory.py still generate a couple issues for me: test_addTimeCycleFeatures (main.TestFeatureMatrixFactory) ... ok test_buildFeatureMatrix_multiClinicalItem (main.TestFeatureMatrixFactory) Test _buildFeatureMatrix(). ... ok test_buildFeatureMatrix_multiFlowsheet (main.TestFeatureMatrixFactory) ... ok test_build_FeatureMatrix_multiLabTest (main.TestFeatureMatrixFactory) ... ERROR test_dbCache (main.TestFeatureMatrixFactory) Test database result caching. ... ok test_loadMapData (main.TestFeatureMatrixFactory) ... ok test_performance (main.TestFeatureMatrixFactory) ... ERROR test_processPatientListInput (main.TestFeatureMatrixFactory) Test processPatientListInput(). ... ok
Traceback (most recent call last): File "medinfo\dataconversion\test\TestFeatureMatrixFactory.py", line 271, in test_build_FeatureMatrix_multiLabTest factory.addLabResultFeatures(labBaseNames, False, preTimeDelta, postTimeDelta) File "C:\HealthRex\CDSS\medinfo\dataconversion\FeatureMatrixFactory.py", line 471, in addLabResultFeatures tempFile = open(tempFileName, "w") IOError: [Errno 22] invalid mode ('w') or filename: 'fmf.patientTNI-CR-LAC-90 days, 0:00:00_0:00:00.tsv'
Traceback (most recent call last): File "medinfo\dataconversion\test\TestFeatureMatrixFactory.py", line 452, in test_performance factory.addLabResultFeatures(["Foo"], preTimeDelta, postTimeDelta) File "C:\HealthRex\CDSS\medinfo\dataconversion\FeatureMatrixFactory.py", line 471, in addLabResultFeatures tempFile = open(tempFileName, "w") IOError: [Errno 22] invalid mode ('w') or filename: 'fmf.patient_Foo_0:00:00_None.tsv'
Ran 8 tests in 3.903s
Looks like you have to change the naming convention for the temporary files (won't allow ":" symbol on my system). You could just use the total_seconds() from the datetime you're using. Also, be sure to have the process clean up (remove) any of these temp files created. For example, I end up with these extra files after running the test: fmf.patient_PerfItem300.tsv fmf.patient_PerfItem400.tsv fmf.patient_PerfItem500.tsv fmf.patient_episodes.tsv fmf.patient_list.tsv patient_list.tsv
SanthoshBala
commented
6 years ago So, turns out that both of the remaining failures (TestFeatureMatrixFactory for you and TestSTRIDEOrderMedConverstion for me) are caused by mismatches between Windows and Linux =)
Specifically: (1) test_build_FeatureMatrix_multiLabTest and test_performance fail for you because ':' is a disallowed filename character on Windows, but not Linux. The temp files do get cleaned up, but not when the test errors (because the cleanup is test-specific vs. tearDown). I just pushed a fix for this. (2) test_dataConversion_denormalized fails for me because STRIDEOrderMedCovnersion.normalizeMixData() calls hash() which is NOT guaranteed to have stable results cross-platform. This fix is to use hashlib instead, but I'm wary of changing code which generates our database without actually updating our production database, and not sure that makes sense right now. I just left this as a TODO for now.
@jonc101 with those two updates, I think we're finally good! Gonna optimistically close, but feel free to reopen if you want to figure out that second hash() issue.
Look for the 1-line python command in setupNotes.txt