search

PlaceAVote / pav-congress-api-bootstrapper

Service for syncing congres data and other manual work
Eclipse Public License 1.0
0 stars 0 forks source link

Problems publish bill updates to redis. Connection problem with carmine library #13

Open johnboy14 opened 8 years ago

johnboy14 commented 8 years ago

This was stripped from the logs in dev and beta environments.

12:19:40.994 INFO  ns:com.pav.congress.bill.bill - Sending to Redis: {:id "sres320-114", :new "PASSED:SIMPLERES", :old "REFERRED"}
12:19:40.995 ERROR ns:com.pav.congress.bill.bill - Failed ending message to Redis
clojure.lang.ExceptionInfo: Carmine connection error
    at clojure.core$ex_info.invoke(core.clj:4593) ~[pav-congress-api-bootstrapper.jar:?]
    at taoensso.carmine.connections$pooled_conn.invoke(connections.clj:201) ~[pav-congress-api-bootstrapper.jar:?]
    at com.pav.congress.bill.bill$index_bill_BANG_$fn__17952.invoke(bill.clj:193) [pav-congress-api-bootstrapper.jar:?]
    at com.pav.congress.bill.bill$index_bill_BANG_.invoke(bill.clj:189) [pav-congress-api-bootstrapper.jar:?]
    at com.pav.congress.bill.bill$persist_bills.invoke(bill.clj:208) [pav-congress-api-bootstrapper.jar:?]
    at com.pav.congress.jobs.bill$batch_and_persist$fn__18802$state_machine__7537__auto____18803$fn__18805$fn__18807.invoke(bill.clj:26) [pav-congress-api-bootstrapper.jar:?]
    at com.pav.congress.jobs.bill$batch_and_persist$fn__18802$state_machine__7537__auto____18803$fn__18805.invoke(bill.clj:14) [pav-congress-api-bootstrapper.jar:?]
    at com.pav.congress.jobs.bill$batch_and_persist$fn__18802$state_machine__7537__auto____18803.invoke(bill.clj:14) [pav-congress-api-bootstrapper.jar:?]
    at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940) [pav-congress-api-bootstrapper.jar:?]
    at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944) [pav-congress-api-bootstrapper.jar:?]
    at com.pav.congress.jobs.bill$batch_and_persist$fn__18802.invoke(bill.clj:14) [pav-congress-api-bootstrapper.jar:?]
    at clojure.lang.AFn.run(AFn.java:22) [pav-congress-api-bootstrapper.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_66-internal]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_66-internal]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_66-internal]
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_66-internal]
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_66-internal]
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_66-internal]
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_66-internal]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_66-internal]
    at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_66-internal]
    at taoensso.carmine.connections$make_new_connection.invoke(connections.clj:73) ~[pav-congress-api-bootstrapper.jar:?]
    at taoensso.carmine.connections$make_connection_factory$reify__11835.makeObject(connections.clj:106) ~[pav-congress-api-bootstrapper.jar:?]
    at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:1041) ~[pav-congress-api-bootstrapper.jar:?]
    at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:357) ~[pav-congress-api-bootstrapper.jar:?]
    at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:279) ~[pav-congress-api-bootstrapper.jar:?]
    at taoensso.carmine.connections.ConnectionPool.get_conn(connections.clj:47) ~[pav-congress-api-bootstrapper.jar:?]
    at taoensso.carmine.connections$pooled_conn.invoke(connections.clj:196) ~[pav-congress-api-bootstrapper.jar:?]
    ... 13 more
sanel commented 8 years ago

Redis went offline here... Is this happening again?

johnboy14 commented 8 years ago

@sanel I have seen it a few times in the past but not recently. At the moment we are running a micro instance in beta environment. I would like to scale up our redis cluster to something abit more substantial.

sanel commented 8 years ago

To mitigate this, proposal is to use fixed queue and temporarily persist failed messages, resending them until next sync is done.