search

OHDSI / MedlineXmlToDatabase

A command line Java application for parsing MEDLINE XML files and inserting the data into a relational database
Apache License 2.0
19 stars 11 forks source link

java.sql.BatchUpdateException #5

Closed phearun008 closed 7 years ago

phearun008 commented 7 years ago

Hello Sir,

Thanks for your working hard with this library. I've been using your library to parse Medline data from XML to RDBMS, and I'm facing some problems like the image shown below. I would like to know what is the problem and if there are any solutions with this?

Thanks,

21640282_870061266494089_742489016_o

schuemie commented 7 years ago

I'm sorry, the error message is not very informative. My guess is something went wrong earlier, causing the rest of the script to fail. Do you have any information on what happens right at the moment when these messages start?

phearun008 commented 7 years ago

I'm sorry, but I saw only this kind of error message. When I parse one XML file, it's OK. But when I try to parse multiple XML files, it shows like this.

vojtechhuser commented 7 years ago

I was able to run it. on windows, and posgres

image

phearun008 commented 7 years ago

Hi @schuemie , Here is the complete error log file.

https://raw.githubusercontent.com/phearun008/pubmed-log/master/log.txt

phearun008 commented 7 years ago

It's working fine now.

lemonysoft commented 6 years ago

Why giving this error? No article was inserted to PostgreSQL.

java.sql.BatchUpdateException: Batch entry 0 DELETE FROM medcit_otherid WHERE pmid = 342622 AND pmid_version = 1 was aborted.  Call getNextException to see the cause.
        at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2740)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1891)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2889)
        at org.ohdsi.databases.ConnectionWrapper.setBatchMode(ConnectionWrapper.java:59)
        at org.ohdsi.medlineXmlToDatabase.MedlineCitationParser.parseAndInjectIntoDB(MedlineCitationParser.java:68)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.analyse(MedlineParserMain.java:65)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.parseFolder(MedlineParserMain.java:54)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.main(MedlineParserMain.java:40)
        at org.ohdsi.medlineXmlToDatabase.MainClass.main(MainClass.java:43)
java.sql.BatchUpdateException: Batch entry 0 DELETE FROM medcit_otherid WHERE pmid = 342623 AND pmid_version = 1 was aborted.  Call getNextException to see the cause.
        at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2740)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1891)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2889)
        at org.ohdsi.databases.ConnectionWrapper.setBatchMode(ConnectionWrapper.java:59)
        at org.ohdsi.medlineXmlToDatabase.MedlineCitationParser.parseAndInjectIntoDB(MedlineCitationParser.java:68)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.analyse(MedlineParserMain.java:65)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.parseFolder(MedlineParserMain.java:54)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.main(MedlineParserMain.java:40)
        at org.ohdsi.medlineXmlToDatabase.MainClass.main(MainClass.java:43)
java.sql.BatchUpdateException: Batch entry 0 DELETE FROM medcit_otherid WHERE pmid = 342625 AND pmid_version = 1 was aborted.  Call getNextException to see the cause.
        at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2740)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1891)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2889)
        at org.ohdsi.databases.ConnectionWrapper.setBatchMode(ConnectionWrapper.java:59)
        at org.ohdsi.medlineXmlToDatabase.MedlineCitationParser.parseAndInjectIntoDB(MedlineCitationParser.java:68)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.analyse(MedlineParserMain.java:65)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.parseFolder(MedlineParserMain.java:54)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.main(MedlineParserMain.java:40)
        at org.ohdsi.medlineXmlToDatabase.MainClass.main(MainClass.java:43)
java.sql.BatchUpdateException: Batch entry 0 DELETE FROM medcit_otherid WHERE pmid = 342626 AND pmid_version = 1 was aborted.  Call getNextException to see the cause.
        at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2740)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1891)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2889)
        at org.ohdsi.databases.ConnectionWrapper.setBatchMode(ConnectionWrapper.java:59)
        at org.ohdsi.medlineXmlToDatabase.MedlineCitationParser.parseAndInjectIntoDB(MedlineCitationParser.java:68)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.analyse(MedlineParserMain.java:65)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.parseFolder(MedlineParserMain.java:54)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.main(MedlineParserMain.java:40)
        at org.ohdsi.medlineXmlToDatabase.MainClass.main(MainClass.java:43)
java.sql.BatchUpdateException: Batch entry 0 DELETE FROM medcit_otherid WHERE pmid = 342624 AND pmid_version = 1 was aborted.  Call getNextException to see the cause.
        at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2740)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1891)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2889)
        at org.ohdsi.databases.ConnectionWrapper.setBatchMode(ConnectionWrapper.java:59)
        at org.ohdsi.medlineXmlToDatabase.MedlineCitationParser.parseAndInjectIntoDB(MedlineCitationParser.java:68)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.analyse(MedlineParserMain.java:65)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.parseFolder(MedlineParserMain.java:54)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.main(MedlineParserMain.java:40)
        at org.ohdsi.medlineXmlToDatabase.MainClass.main(MainClass.java:43)
java.sql.BatchUpdateException: Batch entry 0 DELETE FROM medcit_otherid WHERE pmid = 342627 AND pmid_version = 1 was aborted.  Call getNextException to see the cause.
        at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2740)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1891)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2889)
        at org.ohdsi.databases.ConnectionWrapper.setBatchMode(ConnectionWrapper.java:59)
        at org.ohdsi.medlineXmlToDatabase.MedlineCitationParser.parseAndInjectIntoDB(MedlineCitationParser.java:68)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.analyse(MedlineParserMain.java:65)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.parseFolder(MedlineParserMain.java:54)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.main(MedlineParserMain.java:40)
        at org.ohdsi.medlineXmlToDatabase.MainClass.main(MainClass.java:43)
java.sql.BatchUpdateException: Batch entry 0 DELETE FROM medcit_otherid WHERE pmid = 342628 AND pmid_version = 1 was aborted.  Call getNextException to see the cause.
        at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2740)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1891)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2889)
        at org.ohdsi.databases.ConnectionWrapper.setBatchMode(ConnectionWrapper.java:59)
        at org.ohdsi.medlineXmlToDatabase.MedlineCitationParser.parseAndInjectIntoDB(MedlineCitationParser.java:68)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.analyse(MedlineParserMain.java:65)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.parseFolder(MedlineParserMain.java:54)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.main(MedlineParserMain.java:40)
        at org.ohdsi.medlineXmlToDatabase.MainClass.main(MainClass.java:43)
java.sql.BatchUpdateException: Batch entry 0 DELETE FROM medcit_otherid WHERE pmid = 342630 AND pmid_version = 1 was aborted.  Call getNextException to see the cause.
        at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2740)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1891)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2889)
        at org.ohdsi.databases.ConnectionWrapper.setBatchMode(ConnectionWrapper.java:59)
        at org.ohdsi.medlineXmlToDatabase.MedlineCitationParser.parseAndInjectIntoDB(MedlineCitationParser.java:68)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.analyse(MedlineParserMain.java:65)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.parseFolder(MedlineParserMain.java:54)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.main(MedlineParserMain.java:40)
        at org.ohdsi.medlineXmlToDatabase.MainClass.main(MainClass.java:43)
java.sql.BatchUpdateException: Batch entry 0 DELETE FROM medcit_otherid WHERE pmid = 342629 AND pmid_version = 1 was aborted.  Call getNextException to see the cause.
        at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2740)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1891)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2889)
        at org.ohdsi.databases.ConnectionWrapper.setBatchMode(ConnectionWrapper.java:59)
        at org.ohdsi.medlineXmlToDatabase.MedlineCitationParser.parseAndInjectIntoDB(MedlineCitationParser.java:68)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.analyse(MedlineParserMain.java:65)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.parseFolder(MedlineParserMain.java:54)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.main(MedlineParserMain.java:40)
        at org.ohdsi.medlineXmlToDatabase.MainClass.main(MainClass.java:43)
java.sql.BatchUpdateException: Batch entry 0 DELETE FROM medcit_otherid WHERE pmid = 342632 AND pmid_version = 1 was aborted.  Call getNextException to see the cause.
        at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2740)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1891)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2889)
        at org.ohdsi.databases.ConnectionWrapper.setBatchMode(ConnectionWrapper.java:59)
        at org.ohdsi.medlineXmlToDatabase.MedlineCitationParser.parseAndInjectIntoDB(MedlineCitationParser.java:68)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.analyse(MedlineParserMain.java:65)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.parseFolder(MedlineParserMain.java:54)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.main(MedlineParserMain.java:40)
        at org.ohdsi.medlineXmlToDatabase.MainClass.main(MainClass.java:43)
java.sql.BatchUpdateException: Batch entry 0 DELETE FROM medcit_otherid WHERE pmid = 342633 AND pmid_version = 1 was aborted.  Call getNextException to see the cause.
        at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2740)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1891)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2889)
        at org.ohdsi.databases.ConnectionWrapper.setBatchMode(ConnectionWrapper.java:59)
        at org.ohdsi.medlineXmlToDatabase.MedlineCitationParser.parseAndInjectIntoDB(MedlineCitationParser.java:68)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.analyse(MedlineParserMain.java:65)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.parseFolder(MedlineParserMain.java:54)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.main(MedlineParserMain.java:40)
        at org.ohdsi.medlineXmlToDatabase.MainClass.main(MainClass.java:43)
java.sql.BatchUpdateException: Batch entry 0 DELETE FROM medcit_otherid WHERE pmid = 342631 AND pmid_version = 1 was aborted.  Call getNextException to see the cause.
        at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2740)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1891)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2889)
        at org.ohdsi.databases.ConnectionWrapper.setBatchMode(ConnectionWrapper.java:59)
        at org.ohdsi.medlineXmlToDatabase.MedlineCitationParser.parseAndInjectIntoDB(MedlineCitationParser.java:68)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.analyse(MedlineParserMain.java:65)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.parseFolder(MedlineParserMain.java:54)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.main(MedlineParserMain.java:40)
        at org.ohdsi.medlineXmlToDatabase.MainClass.main(MainClass.java:43)
java.sql.BatchUpdateException: Batch entry 0 DELETE FROM medcit_otherid WHERE pmid = 342634 AND pmid_version = 1 was aborted.  Call getNextException to see the cause.
        at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2740)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1891)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2889)
        at org.ohdsi.databases.ConnectionWrapper.setBatchMode(ConnectionWrapper.java:59)
        at org.ohdsi.medlineXmlToDatabase.MedlineCitationParser.parseAndInjectIntoDB(MedlineCitationParser.java:68)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.analyse(MedlineParserMain.java:65)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.parseFolder(MedlineParserMain.java:54)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.main(MedlineParserMain.java:40)
        at org.ohdsi.medlineXmlToDatabase.MainClass.main(MainClass.java:43)
java.sql.BatchUpdateException: Batch entry 0 DELETE FROM medcit_otherid WHERE pmid = 342635 AND pmid_version = 1 was aborted.  Call getNextException to see the cause.
        at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2740)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1891)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2889)
        at org.ohdsi.databases.ConnectionWrapper.setBatchMode(ConnectionWrapper.java:59)
        at org.ohdsi.medlineXmlToDatabase.MedlineCitationParser.parseAndInjectIntoDB(MedlineCitationParser.java:68)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.analyse(MedlineParserMain.java:65)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.parseFolder(MedlineParserMain.java:54)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.main(MedlineParserMain.java:40)
schuemie commented 6 years ago

I'm afraid these error messages aren't very informative. Most likely something else has gone wrong before these commands, and the connection has been terminated.

I've just created a new release that should provide better error messaging. Could you try that one?

lemonysoft commented 6 years ago

Here are the errors printed on screen:

[root@dhpc09 administrator]# sudo java -Xmx5000m -jar /var/aliriza/xml2db/MedlineXmlToDatabase.jar -ini /var/aliriza/xml2db/MedlineXmlToDatabase.ini -parse
Processing pubmed18n0001.xml.gz
Processing pubmed18n0002.xml.gz
Error: Batch entry 67 INSERT INTO medcit (owner,art_vernaculartitle,art_journal_journalissue_pubdate_year,art_journal_isoabbreviation,datecompleted_year,art_journal_journalissue_pubdate_month,datecompleted_month,art_authorlist_completeyn,medlinejournalinfo_medlineta,art_journal_title,art_pagination_medlinepgn,daterevised_year,art_journal_journalissue_volume,art_pubmodel,art_journal_journalissue_citedmedium,status,daterevised_day,daterevised_month,datecompleted_day,art_arttitle,art_journal_journalissue_issue,medlinejournalinfo_country,medlinejournalinfo_nlmuniqueid,pmid_version,art_journal_issn,pmid,art_journal_issn_issntype,medlinejournalinfo_issnlinking) VALUES ('NLM','Nachweis krebshemmender Eigenschaften einer stark immunstimulierenden Verbindung kleiner Molekülmasse. Versuche am immunlabilen DS-Karzinosarkom im Vergleich mit Ifosfamid. Stimulierung der körpereigenen Abwehr über etwa 20 Tage durch BA 1, einen N-(2-Cyanthylen)-harnstoff. Neue prophylaktische Möglichkeiten','1975','Arzneimittelforschung','1976','Sep','01','Y','Arzneimittelforschung','Arzneimittel-Forschung','1369-79','2017','25','Print','Print','MEDLINE','22','03','26','[Demonstration of tumor inhibiting properties of a strongly immunostimulating low-molecular weight substance. Comparative studies with ifosfamide on the immuno-labile DS carcinosarcoma. Stimulation of the autoimmune activity for approx. 20 days by BA 1, a N-(2-cyanoethylene)-urea. Novel prophylactic possibilities].','9','Germany','0372660','1','0004-4172','22','Print','0004-4172') was aborted.  Call getNextException to see the cause.
java.sql.BatchUpdateException: Batch entry 67 INSERT INTO medcit (owner,art_vernaculartitle,art_journal_journalissue_pubdate_year,art_journal_isoabbreviation,datecompleted_year,art_journal_journalissue_pubdate_month,datecompleted_month,art_authorlist_completeyn,medlinejournalinfo_medlineta,art_journal_title,art_pagination_medlinepgn,daterevised_year,art_journal_journalissue_volume,art_pubmodel,art_journal_journalissue_citedmedium,status,daterevised_day,daterevised_month,datecompleted_day,art_arttitle,art_journal_journalissue_issue,medlinejournalinfo_country,medlinejournalinfo_nlmuniqueid,pmid_version,art_journal_issn,pmid,art_journal_issn_issntype,medlinejournalinfo_issnlinking) VALUES ('NLM','Nachweis krebshemmender Eigenschaften einer stark immunstimulierenden Verbindung kleiner Molekülmasse. Versuche am immunlabilen DS-Karzinosarkom im Vergleich mit Ifosfamid. Stimulierung der körpereigenen Abwehr über etwa 20 Tage durch BA 1, einen N-(2-Cyanthylen)-harnstoff. Neue prophylaktische Möglichkeiten','1975','Arzneimittelforschung','1976','Sep','01','Y','Arzneimittelforschung','Arzneimittel-Forschung','1369-79','2017','25','Print','Print','MEDLINE','22','03','26','[Demonstration of tumor inhibiting properties of a strongly immunostimulating low-molecular weight substance. Comparative studies with ifosfamide on the immuno-labile DS carcinosarcoma. Stimulation of the autoimmune activity for approx. 20 days by BA 1, a N-(2-cyanoethylene)-urea. Novel prophylactic possibilities].','9','Germany','0372660','1','0004-4172','22','Print','0004-4172') was aborted.  Call getNextException to see the cause.
        at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2740)
        at org.postgresql.core.v3.QueryExecutorImpl$1.handleError(QueryExecutorImpl.java:461)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1891)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2889)
        at org.ohdsi.databases.ConnectionWrapper.setBatchMode(ConnectionWrapper.java:59)
        at org.ohdsi.medlineXmlToDatabase.MedlineCitationParser.parseAndInjectIntoDB(MedlineCitationParser.java:68)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.analyse(MedlineParserMain.java:65)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.parseFolder(MedlineParserMain.java:54)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.main(MedlineParserMain.java:40)
        at org.ohdsi.medlineXmlToDatabase.MainClass.main(MainClass.java:43)
Error: ERROR: value too long for type character varying(255)
org.postgresql.util.PSQLException: ERROR: value too long for type character varying(255)
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2889)
        at org.ohdsi.databases.ConnectionWrapper.setBatchMode(ConnectionWrapper.java:59)
        at org.ohdsi.medlineXmlToDatabase.MedlineCitationParser.parseAndInjectIntoDB(MedlineCitationParser.java:68)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.analyse(MedlineParserMain.java:65)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.parseFolder(MedlineParserMain.java:54)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.main(MedlineParserMain.java:40)
        at org.ohdsi.medlineXmlToDatabase.MainClass.main(MainClass.java:43)
Exception in thread "main" java.lang.RuntimeException: Error executing batch data
        at org.ohdsi.databases.ConnectionWrapper.setBatchMode(ConnectionWrapper.java:74)
        at org.ohdsi.medlineXmlToDatabase.MedlineCitationParser.parseAndInjectIntoDB(MedlineCitationParser.java:68)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.analyse(MedlineParserMain.java:65)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.parseFolder(MedlineParserMain.java:54)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.main(MedlineParserMain.java:40)
        at org.ohdsi.medlineXmlToDatabase.MainClass.main(MainClass.java:43)

My "ini" file: DATA_SOURCE_TYPE = POSTGRESQL # POSTGRESQL, or MSSQL DOMAIN = # Domain when using windows authentication (MSSQL) USER = pubmeduser # User name for the database PASSWORD = asdf1234 # Password for the database SCHEMA = public # Name of the data schema used SERVER = 127.0.0.1/pubmeddb # Name or address of the server. For Postgres, add the database name XML_FOLDER = /var/pubmed2018/baseline # Location of the Medline XML files (GZipped) MESH_XML_FOLDER = /var/pubmed2018/baseline # Location of the MeSH XML files (GZipped)

schuemie commented 6 years ago

Hmmm, the error message says a value is more than 255 characters, and therefore cannot fit in a VARCHAR(255). There are two values with more than 255 characters in that statement: art_vernaculartitle and art_arttitle (both in the medcit table).

The odd thing is that if you ran the analysis on all XML files, it should already have figured out those fields should be of type TEXT, not VARCHAR. I ran the analysis on a few XML files and already it figured this out.

Could you check what the type of those two fields is? If they are not of type TEXT, please change them to TEXT.

lemonysoft commented 6 years ago

I changed them to TEXT. I think this issue was solved. Now tool gives no error. Thanks.

Processing pubmed18n0001.xml.gz
Processing pubmed18n0002.xml.gz
10000 of 30000 (110390ms since start)
20000 of 30000 (217774ms since start)
lemonysoft commented 6 years ago

When parsing, I have another problem shown below. I already deleted all records before parsing.

Is it possible to solve this?

# sudo java -Xmx5000m -jar /var/aliriza/xml2db/MedlineXmlToDatabase.jar -ini /var/aliriza/xml2db/MedlineXmlToDatabase.ini -parse > /var/aliriza/xml2db/parse_log_2018.txt
Error: Batch entry 40 INSERT INTO medcit_keywordlist_keyword (value,pmid_version,majortopicyn,pmid,medcit_keywordlist_keyword_order) VALUES ('Abortion, Induced--legal aspects','1','Y','274614','1') was aborted.  Call getNextException to see the cause.
java.sql.BatchUpdateException: Batch entry 40 INSERT INTO medcit_keywordlist_keyword (value,pmid_version,majortopicyn,pmid,medcit_keywordlist_keyword_order) VALUES ('Abortion, Induced--legal aspects','1','Y','274614','1') was aborted.  Call getNextException to see the cause.
        at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2740)
        at org.postgresql.core.v3.QueryExecutorImpl$1.handleError(QueryExecutorImpl.java:461)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1891)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2889)
        at org.ohdsi.databases.ConnectionWrapper.setBatchMode(ConnectionWrapper.java:59)
        at org.ohdsi.medlineXmlToDatabase.MedlineCitationParser.parseAndInjectIntoDB(MedlineCitationParser.java:68)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.analyse(MedlineParserMain.java:65)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.parseFolder(MedlineParserMain.java:54)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.main(MedlineParserMain.java:40)
        at org.ohdsi.medlineXmlToDatabase.MainClass.main(MainClass.java:43)
Error: ERROR: duplicate key value violates unique constraint "medcit_keywordlist_keyword_pkey"
  Detail: Key (pmid, pmid_version, medcit_keywordlist_keyword_order)=(274614, 1, 1) already exists.
org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "medcit_keywordlist_keyword_pkey"
  Detail: Key (pmid, pmid_version, medcit_keywordlist_keyword_order)=(274614, 1, 1) already exists.
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2889)
        at org.ohdsi.databases.ConnectionWrapper.setBatchMode(ConnectionWrapper.java:59)
        at org.ohdsi.medlineXmlToDatabase.MedlineCitationParser.parseAndInjectIntoDB(MedlineCitationParser.java:68)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.analyse(MedlineParserMain.java:65)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.parseFolder(MedlineParserMain.java:54)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.main(MedlineParserMain.java:40)
        at org.ohdsi.medlineXmlToDatabase.MainClass.main(MainClass.java:43)
Exception in thread "main" java.lang.RuntimeException: Error executing batch data
        at org.ohdsi.databases.ConnectionWrapper.setBatchMode(ConnectionWrapper.java:74)
        at org.ohdsi.medlineXmlToDatabase.MedlineCitationParser.parseAndInjectIntoDB(MedlineCitationParser.java:68)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.analyse(MedlineParserMain.java:65)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.parseFolder(MedlineParserMain.java:54)
        at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.main(MedlineParserMain.java:40)
        at org.ohdsi.medlineXmlToDatabase.MainClass.main(MainClass.java:43)
lemonysoft commented 6 years ago

I ran tool again, still the same:

Processing pubmed18n0010.xml.gz 10000 of 30000 (95423ms since start) 20000 of 30000 (193157ms since start) Error: Batch entry 40 INSERT INTO medcit_keywordlist_keyword (value,pmid_version,majortopicyn,pmid,medcit_keywordlist_keyword_order) VALUES ('Abortion, Induced--legal aspects','1','Y','274614','1') was aborted. Call getNextException to see the cause. java.sql.BatchUpdateException: Batch entry 40 INSERT INTO medcit_keywordlist_keyword (value,pmid_version,majortopicyn,pmid,medcit_keywordlist_keyword_order) VALUES ('Abortion, Induced--legal aspects','1','Y','274614','1') was aborted. Call getNextException to see the cause. at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2740) at org.postgresql.core.v3.QueryExecutorImpl$1.handleError(QueryExecutorImpl.java:461) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1891) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2889) at org.ohdsi.databases.ConnectionWrapper.setBatchMode(ConnectionWrapper.java:59) at org.ohdsi.medlineXmlToDatabase.MedlineCitationParser.parseAndInjectIntoDB(MedlineCitationParser.java:68) at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.analyse(MedlineParserMain.java:65) at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.parseFolder(MedlineParserMain.java:54) at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.main(MedlineParserMain.java:40) at org.ohdsi.medlineXmlToDatabase.MainClass.main(MainClass.java:43) Error: ERROR: duplicate key value violates unique constraint "medcit_keywordlist_keyword_pkey" Detail: Key (pmid, pmid_version, medcit_keywordlist_keyword_order)=(274614, 1, 1) already exists. org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "medcit_keywordlist_keyword_pkey" Detail: Key (pmid, pmid_version, medcit_keywordlist_keyword_order)=(274614, 1, 1) already exists. at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2889) at org.ohdsi.databases.ConnectionWrapper.setBatchMode(ConnectionWrapper.java:59) at org.ohdsi.medlineXmlToDatabase.MedlineCitationParser.parseAndInjectIntoDB(MedlineCitationParser.java:68) at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.analyse(MedlineParserMain.java:65) at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.parseFolder(MedlineParserMain.java:54) at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.main(MedlineParserMain.java:40) at org.ohdsi.medlineXmlToDatabase.MainClass.main(MainClass.java:43) Exception in thread "main" java.lang.RuntimeException: Error executing batch data at org.ohdsi.databases.ConnectionWrapper.setBatchMode(ConnectionWrapper.java:74) at org.ohdsi.medlineXmlToDatabase.MedlineCitationParser.parseAndInjectIntoDB(MedlineCitationParser.java:68) at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.analyse(MedlineParserMain.java:65) at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.parseFolder(MedlineParserMain.java:54) at org.ohdsi.medlineXmlToDatabase.MedlineParserMain.main(MedlineParserMain.java:40) at org.ohdsi.medlineXmlToDatabase.MainClass.main(MainClass.java:43)

schuemie commented 6 years ago

The problem appears to be a duplicate primary key. Have you tried truncating all tables and just starting from scratch?

lemonysoft commented 6 years ago

Just 45min ago, I dropped database. Now tool analysing. I will report it to you.

31 Oca 2018 16:08 tarihinde "Martijn Schuemie" notifications@github.com yazdı:

The problem appears to be a duplicate primary key. Have you tried truncating all tables and just starting from scratch?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/OHDSI/MedlineXmlToDatabase/issues/5#issuecomment-361927021, or mute the thread https://github.com/notifications/unsubscribe-auth/AcuNU2RxohX_-nOkzTniLTY2M_jblsGaks5tQGXmgaJpZM4PZJL- .

lemonysoft commented 6 years ago

Bad news.

` Processing pubmed18n0778.xml.gz

Processing pubmed18n0779.xml.gz

Processing pubmed18n0780.xml.gz

Exception in thread "Thread-0" java.lang.OutOfMemoryError: GC overhead limit exceeded at com.sun.org.apache.xerces.internal.dom.DeferredDocumentImpl.createChunk(DeferredDocumentImpl.java:1935) at com.sun.org.apache.xerces.internal.dom.DeferredDocumentImpl.ensureCapacity(DeferredDocumentImpl.java:1840) at com.sun.org.apache.xerces.internal.dom.DeferredDocumentImpl.createNode(DeferredDocumentImpl.java:1854) at com.sun.org.apache.xerces.internal.dom.DeferredDocumentImpl.setDeferredAttribute(DeferredDocumentImpl.java:462) at com.sun.org.apache.xerces.internal.parsers.AbstractDOMParser.startElement(AbstractDOMParser.java:1110) at com.sun.org.apache.xerces.internal.impl.dtd.XMLDTDValidator.startElement(XMLDTDValidator.java:745) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:1359) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2784) at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:842) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771) at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141) at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243) at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339) at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121) at org.ohdsi.medlineXmlToDatabase.XMLFileIterator$DecompressAndParseThread.process(XMLFileIterator.java:130) at org.ohdsi.utilities.concurrency.BatchProcessingThread.run(BatchProcessingThread.java:42)`

lemonysoft commented 6 years ago

I dropped all db and started with 20GB memory for java. Then problem gone! Java sometimes makes me so angry. Sorry for my disturbance...