search

msathis / SQLToNoSQLImporter

Solr like data import handler to migrate data from sql systems to nosql
GNU Lesser General Public License v3.0
32 stars 14 forks source link

OutOfMemory java heap space #16

Open cyrishikesh opened 5 years ago

cyrishikesh commented 5 years ago

@msathis What is the maximum no. of rows, it can process. My database is pretty large I am getting this error. But for small database it is working fine. Is it pssible to read data in small batches and write to mongodb , So we can process large size database(approax million rows in tables).

INFO: Discovered cluster type of STANDALONE

15:41:30,728 INFO DataImporter:146 - Data Configuration loaded successfully

15:41:30,736 INFO DataImporter:187 - BUILT DOCS

java.lang.OutOfMemoryError:` Java heap space

16:00:29,679` ERROR DataImporter:193 - * Data import failed. **

at sun.text.resources.FormatData.getContents(FormatData.java:112)

Reason is :

at sun.util.resources.ParallelListResourceBundle.loadLookupTablesIfNecessary(ParallelListResourceBundle.java:168)

at sun.util.resources.ParallelListResourceBundle.handleKeySet(ParallelListResourceBundle.java:134)

at sun.util.resources.ParallelListResourceBundle.keySet(ParallelListResourceBundle.java:143)

at sun.util.resources.ParallelListResourceBundle.containsKey(ParallelListResourceBundle.java:129)

at sun.util.resources.ParallelListResourceBundle$KeySet.contains(ParallelListResourceBundle.java:208)

at sun.util.resources.ParallelListResourceBundle.containsKey(ParallelListResourceBundle.java:129)

at sun.util.resources.ParallelListResourceBundle$KeySet.contains(ParallelListResourceBundle.java:208)

at sun.util.resources.ParallelListResourceBundle.containsKey(ParallelListResourceBundle.java:129)

at java.text.DateFormatSymbols.initializeData(DateFormatSymbols.java:716)

at java.text.DateFormatSymbols.(DateFormatSymbols.java:145)

at sun.util.locale.provider.DateFormatSymbolsProviderImpl.getInstance(DateFormatSymbolsProviderImpl.java:85)

at java.text.DateFormatSymbols.getProviderInstance(DateFormatSymbols.java:364)

at java.text.DateFormatSymbols.getInstanceRef(DateFormatSymbols.java:354)

at java.text.SimpleDateFormat.(SimpleDateFormat.java:603)

at java.text.SimpleDateFormat.(SimpleDateFormat.java:580)

at net.sathis.export.sql.model.FieldTypeParser.dateToString(FieldTypeParser.java:15)

at net.sathis.export.sql.DocBuilder.convertFieldType(DocBuilder.java:234)

at net.sathis.export.sql.DocBuilder.extractFieldValue(DocBuilder.java:219)

at net.sathis.export.sql.DocBuilder.getSingleValuedEntity(DocBuilder.java:204)

at net.sathis.export.sql.DocBuilder.getFields(DocBuilder.java:131)

at net.sathis.export.sql.DocBuilder.getFields(DocBuilder.java:158)

at net.sathis.export.sql.DocBuilder.getFields(DocBuilder.java:158)

at net.sathis.export.sql.DocBuilder.getFields(DocBuilder.java:158)

at net.sathis.export.sql.DocBuilder.execute(DocBuilder.java:114)

at net.sathis.export.sql.DataImporter.doFullImport(DataImporter.java:188)

at net.sathis.export.sql.DataImporter.doDataImport(DataImporter.java:96)

at net.sathis.export.sql.SQLToNoSQLImporter.main(SQLToNoSQLImporter.java:20)

msathis commented 5 years ago

Out of memory error shouldnt happen normally unless your Heap size is too small. Can you please try increasing it a bit further?

msathis commented 5 years ago

Actually, it reads in small batches only. I already tested this with 100 million rows earlier. Not sure why it won't work.

cyrishikesh commented 5 years ago

Actually, it reads in small batches only. I already tested this with 100 million rows earlier. Not sure why it won't work.

Actually for one table, it is running fine for 10 milllion rows. But I am embedding 5 tables into one collections. Maybe it's because of internal calls. How many internal calls it can handle??