jprante / elasticsearch-jdbc

JDBC importer for Elasticsearch
Apache License 2.0
2.84k stars 710 forks source link

java.lang.OutOfMemoryError: unable to create new native thread #826

Open RebirthOfSilence opened 8 years ago

RebirthOfSilence commented 8 years ago

{ "type" : "jdbc", "jdbc" : { "bulktimeout" : "10s", "index" : "**", "bulk_size" : "100", "flushinterval" : "1s", "type" : "**", "max_bulkrequests" : "30", "url" : "jdbc:mysql://**:3306", "sql" : [ { "statement" : "select a.car_id _id,a.car_id,a.seller_id,a.model_id,a.car_area,a.car_status,UNIX_TIMESTAMP(a.reach_date)_1000 AS reach_date,a.seller_price,a.invoice_price,a.car_unique,a.img_url,a.is_deleted,a.inner_id,a.outer_id,a.outer_color_name,a.inner_color_name,a.send_city,a.benchmark,a.fuel,a.quantity,a.insurance,a.configure,a.remark,a.sanbao,a.formalities_status,a.custom_model,a.custom_formality,a.custom_area,a.spec,a.type,a.brand_id,a.series_id,UNIX_TIMESTAMP(a.gmt_create)_1000 AS gmt_create,UNIX_TIMESTAMP(a.gmt_modified)_1000 AS modify,a.config_price,a.price_adjust_way,a.price_adjust_val,a.is_dianyi,a.status AS sku_status,a.is_induce,a.index_word,a.car_source,a.ext_id,a.guide_price,b.corp_name,b.is_verify_merchant,c.model_name,c.standard,c.displacement,c.drive,c.seat,c.year,c.status,d.series_name,e.alias from db_maihaoche.mhc_b2b_car a left join db_b2b.b2b_partner b on a.seller_id=b.id left join db_maihaoche.mhc_model c on a.model_id=c.model_id left join db_maihaoche.mhc_series d on a.series_id=d.series_id left join db_maihaoche.mhc_synonym e on d.series_id=e.series_id where a.gmt_create > ? or GREATEST(a.gmt_modified, b.gmt_modified, c.gmt_modified, d.gmt_modified, e.gmtmodified) > ?", "parameter" : [ "$metrics.lastexecutionstart", "$metrics.lastexecutionstart" ] } ], "threadpoolsize" : "1", "schedule" : "0/1 0-59 0-23 ? * *", "password" : "**", "statefile" : "S-import-carsource.json", "interval" : "0s", "metrics" : { "lastexecutionstart" : "2016-05-24T05:53:15.027Z", "lastexecutionend" : "2016-05-24T05:53:15.724Z", "counter" : "9525" }, "user" : "**" } }

I do schedule task to collect incremental data from mysql,but soon the jdbc-import program give me exception like“java.lang.OutOfMemeoryError: unable to create new native thread",how to resolve the problem? PS:the elasticsearch-jdbc version is:2.3.1,and the elasticsearch too

jprante commented 8 years ago

As first aid, remove

"flush_interval" : "1s"
RebirthOfSilence commented 8 years ago

I removed that,I monitor the java program threads‘s increasing like this: 41 30 30 30 30 30 30 30 30 30 32 32 32 32 32 32 32 32 32 33 34 34 34 34 34 34 34 34 34 46 36 36 36 36 36 36 36 36 36 48 38 38 38 38 38 38 38 38 38 51 40 40 40 40 40 40 40 40 40 52 42 42 42 42 42 42 42 42 42 54 44 44 44 44 44 44 44 44 44 46 46 46 46 46 46 46 46 46 57 48 48 48 48 48 48 48 48 48 60 50 50 50 50 50 50 50 50 50 63 52 52 52 52 52 52 52 52 52 65 54 54 54 54 54 54 54 54 54 67 56 56 56 56 56 56 56 56 56 58 58 java thread continue increasing,then one day it will give exception like OOM. I want to keep the java threads in a balanced num,use fixed ThreadPoolExecutor is a best way,I hope you can use it,not new a java thread to execute a coming task everytime,also I found many threads status keep WAITING or TIMED_WAITING,that confused me!

rainkong commented 8 years ago

Do you resolve this issue ? I also met this issue.

nidonglin commented 8 years ago

Do you resolve this issue ? I also met this issue.