search

contribsys / faktory

Language-agnostic persistent background job server
https://contribsys.com/faktory/
Other
5.73k stars 227 forks source link

Jobs getting enqueued but not processed #373

Closed bploetz closed 2 years ago

bploetz commented 3 years ago

Docker container contribsys/faktory:1.4.1

faktory_worker_go v1.4.0

    mgr := worker.NewManager()

    // use up to N goroutines to execute jobs
    concurrencyString := os.Getenv("NUM_GO_ROUTINES")
    concurrency, err := strconv.Atoi(concurrencyString)
    if err != nil {
        panic("NUM_GO_ROUTINES is not a number")
    }
    mgr.Concurrency = concurrency

    pool, nperr := faktory.NewPool(concurrency + 2)
    if nperr != nil {
        log.Panicf("Couldn't create Faktory connection pool: %+v", nperr)
    }
    mgr.Pool = pool

    receiptClient, cerr := mgr.Pool.Get()
    if cerr != nil {
        log.Panicf("Couldn't get Faktory client from pool: %+v", cerr)
    }

    // register job types and the function to execute them
    mgr.Register("foo", fooProcessor.Process)
        // etc.....

    // pull jobs from these queues, in this order of precedence
    mgr.ProcessStrictPriorityQueues("foo", "bar", "baz", "etc")

    // Start processing jobs, this method does not return
    fmt.Println("starting job workers")
    mgr.Run()

Are you using an old version? Have you checked the changelogs to see if your issue has been fixed in a later version?

https://github.com/contribsys/faktory/blob/master/Changes.md https://github.com/contribsys/faktory/blob/master/Pro-Changes.md https://github.com/contribsys/faktory/blob/master/Ent-Changes.md

Yes, using an old version, nothing in changelog of note as far as I can tell.

We've been running Faktory in production with no issues for over a year. All of a sudden today, we got an alert that the number of enqueued jobs was high. We saw the Enqueued size keep growing but nothing was getting processed. I restarted the worker process, which processed through the backlog, then everything got stuck again.

I sent the TTIN signal as noted here contribsys/faktory_worker_go#33 but I can't make heads or tails of the thread dump.

We killed the Faktory server itself and that seems to have fixed the issue. Unfortunately I did not get a thread dump on the server itself, but it had been running since 2020-08-28 09:49:22 -0400.

Unfortunately when you kill the server, you lose all of the enqueued jobs. I copied all of the enqueued jobs and their arguments from the Faktory UI and re-enqueued them manually in the new Faktory server, but is there a better way to gracefully shut down Faktory and bring it back up with the same data? Ironically all of the jobs in the Dead queue survived into the new instance and the old metrics (number of jobs run, etc), so I assume that means it is using the mounted volume into the container correctly for persisting redis data?

mperham commented 3 years ago

There's not a lot to go on here. I need more data.

  1. Can you show me the TTIN output?
  2. Can you update to the latest version and see if that helps?
  3. Can you show me the contents of the /debug page when it is misbehaving?
bploetz commented 3 years ago

worker_thread_dump.txt

@mperham

  1. worker TTIN thread dump attached.
  2. Not quickly, no
  3. I didn't happen to look at the /debug page when this was happening, and like I said I killed the server, so I can't go back in time unfortunately. I can't force the server/worker into this state at will, so if it happens to happen again I'll grab the /debug page output if/when it happens.
mperham commented 3 years ago

The only unusual thing I see there is a panic recovery (maybe your NewRelic layer tries to recover?):

panic: runtime error: slice bounds out of range [91:80] [recovered]
    panic: runtime error: slice bounds out of range [91:80]
goroutine 67 [running]:
github.com/newrelic/go-agent/v3/newrelic.(*thread).End(0xc000483b10, 0xcf7a40, 0xc000388000, 0x0, 0x0)
    /go/pkg/mod/github.com/newrelic/go-agent/v3@v3.9.0/newrelic/internal_txn.go:475 +0x163c
github.com/newrelic/go-agent/v3/newrelic.(*Transaction).End(0xc0005ed4e0)
    /go/pkg/mod/github.com/newrelic/go-agent/v3@v3.9.0/newrelic/transaction.go:41 +0x67
panic(0xcf7a40, 0xc000388000)
    /usr/local/go/src/runtime/panic.go:679 +0x1b2
bufio.(*Reader).ReadSlice(0xc0000a8f00, 0xa, 0xc00069aa78, 0x4c55c8, 0x0, 0x0, 0xc00069ad70)
    /usr/local/go/src/bufio/bufio.go:334 +0x22d
bufio.(*Reader).ReadBytes(0xc0000a8f00, 0xa, 0x0, 0x0, 0x0, 0x0, 0xc0004a12ec)
    /usr/local/go/src/bufio/bufio.go:437 +0x73
github.com/contribsys/faktory/client.readResponse(0xc0000a8f00, 0x2, 0x0, 0x0, 0xc00069acd0, 0x8de4ca)
    /go/pkg/mod/github.com/contribsys/faktory@v1.3.0-1/client/client.go:493 +0x72
github.com/contribsys/faktory/client.ok(0xc0000a8f00, 0x0, 0x0)
    /go/pkg/mod/github.com/contribsys/faktory@v1.3.0-1/client/client.go:456 +0x2f
github.com/contribsys/faktory/client.(*Client).ok(0xc000206100, 0xc0000a8f00, 0xd64a71, 0x4)
    /go/pkg/mod/github.com/contribsys/faktory@v1.3.0-1/client/client.go:394 +0x2b
github.com/contribsys/faktory/client.(*Client).Push(0xc000206100, 0xc0006e5720, 0xc000b8b070, 0x1)
    /go/pkg/mod/github.com/contribsys/faktory@v1.3.0-1/client/client.go:257 +0xd2
github.com/merryfield/reward-service/processors.enqueueUpdateProfileJob(0xc000414920, 0xc0006f2c30, 0x24, 0xfd6040, 0xc000206100, 0x0, 0x0)
    /opt/reward-service/processors/receipt_processing_helpers.go:697 +0x1da
github.com/merryfield/reward-service/processors.(*DigitalReceiptProcessor).Process(0xc00023e240, 0xfe7b00, 0xc0002e1c80, 0xc0000f4900, 0x1, 0x4, 0x0, 0x0)
    /opt/reward-service/processors/digital_receipt.go:217 +0x23d3

Otherwise I don't see any type of smoking gun. You are using Faktory client 1.3.0 but I don't see any bug fixes that would definitely help. I would recommend upgrading both Faktory and the client version if possible just so you are on the latest code and we can rule out any old, fixed bugs.

bploetz commented 2 years ago

@mperham this has happened twice in the last week. Output from the /debug page:

Debugging:

Screen Shot 2022-04-13 at 12 04 31 PM

Redis info:

# Server
redis_version:5.0.7
redis_git_sha1:bed89672
redis_git_dirty:0
redis_build_id:575dc7b6da705497
redis_mode:standalone
os:Linux 4.14.158-129.185.amzn2.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:9.2.0
process_id:10
run_id:840f9e5c395f841819460a39d1c9c8f9cf38c7f8
tcp_port:0
uptime_in_seconds:22380553
uptime_in_days:259
hz:10
configured_hz:10
lru_clock:5698488
executable:/usr/bin/redis-server
config_file:/tmp/redis.conf

# Clients
connected_clients:3
client_recent_max_input_buffer:2
client_recent_max_output_buffer:0
blocked_clients:2

# Memory
used_memory:789480
used_memory_human:770.98K
used_memory_rss:3846144
used_memory_rss_human:3.67M
used_memory_peak:3098629
used_memory_peak_human:2.96M
used_memory_peak_perc:25.48%
used_memory_overhead:699998
used_memory_startup:550324
used_memory_dataset:89482
used_memory_dataset_perc:37.42%
allocator_allocated:755718
allocator_active:3808256
allocator_resident:3808256
total_system_memory:8053796864
total_system_memory_human:7.50G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:5.04
allocator_frag_bytes:3052538
allocator_rss_ratio:1.00
allocator_rss_bytes:0
rss_overhead_ratio:1.01
rss_overhead_bytes:37888
mem_fragmentation_ratio:5.09
mem_fragmentation_bytes:3090426
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:83538
mem_aof_buffer:0
mem_allocator:libc
active_defrag_running:0
lazyfree_pending_objects:0

# Persistence
loading:0
rdb_changes_since_last_save:2
rdb_bgsave_in_progress:0
rdb_last_save_time:1649865643
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:487424
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0

# Stats
total_connections_received:3015
total_commands_processed:231609196
instantaneous_ops_per_sec:1
total_net_input_bytes:31338644618
total_net_output_bytes:1584414740
instantaneous_input_kbps:0.17
instantaneous_output_kbps:0.01
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:5367534
keyspace_misses:7711769
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:116
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0

# Replication
role:master
connected_slaves:0
master_replid:a5f09e94d96c9a74e05086116c8f27b5e64e4671
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:10506.235012
used_cpu_user:9990.253246
used_cpu_sys_children:75.834772
used_cpu_user_children:403.868009

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=1243,expires=0,avg_ttl=0

Disk Usage:

> df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                  98.3G      4.8G     93.1G   5% /
tmpfs                    64.0M         0     64.0M   0% /dev
tmpfs                     3.8G         0      3.8G   0% /sys/fs/cgroup
/dev/nvme0n1p1           98.3G      4.8G     93.1G   5% /etc/resolv.conf
/dev/nvme0n1p1           98.3G      4.8G     93.1G   5% /etc/hostname
/dev/nvme0n1p1           98.3G      4.8G     93.1G   5% /etc/hosts
shm                      64.0M         0     64.0M   0% /dev/shm
fs-006ac882.efs.us-east-1.amazonaws.com:/
                          8.0E         0      8.0E   0% /var/lib/faktory
tmpfs                     3.8G         0      3.8G   0% /proc/acpi
tmpfs                    64.0M         0     64.0M   0% /proc/kcore
tmpfs                    64.0M         0     64.0M   0% /proc/keys
tmpfs                    64.0M         0     64.0M   0% /proc/latency_stats
tmpfs                    64.0M         0     64.0M   0% /proc/timer_list
tmpfs                    64.0M         0     64.0M   0% /proc/sched_debug
tmpfs                     3.8G         0      3.8G   0% /sys/firmware
mperham commented 2 years ago

You’re on a version, 1.4.1, that is a bit old. Can you upgrade to see if that fixes the issue?

On Wed, Apr 13, 2022 at 09:05 Brian Ploetz @.***> wrote:

@mperham https://github.com/mperham this has happened twice in the last week. Output from the /debug page:

Debugging: [image: Screen Shot 2022-04-13 at 12 04 31 PM] https://user-images.githubusercontent.com/92764/163222819-13280ed5-de19-4034-b957-e51822675d1d.png

Redis info:

Server

redis_version:5.0.7 redis_git_sha1:bed89672 redis_git_dirty:0 redis_build_id:575dc7b6da705497 redis_mode:standalone os:Linux 4.14.158-129.185.amzn2.x86_64 x86_64 arch_bits:64 multiplexing_api:epoll atomicvar_api:atomic-builtin gcc_version:9.2.0 process_id:10 run_id:840f9e5c395f841819460a39d1c9c8f9cf38c7f8 tcp_port:0 uptime_in_seconds:22380553 uptime_in_days:259 hz:10 configured_hz:10 lru_clock:5698488 executable:/usr/bin/redis-server config_file:/tmp/redis.conf

Clients

connected_clients:3 client_recent_max_input_buffer:2 client_recent_max_output_buffer:0 blocked_clients:2

Memory

used_memory:789480 used_memory_human:770.98K used_memory_rss:3846144 used_memory_rss_human:3.67M used_memory_peak:3098629 used_memory_peak_human:2.96M used_memory_peak_perc:25.48% used_memory_overhead:699998 used_memory_startup:550324 used_memory_dataset:89482 used_memory_dataset_perc:37.42% allocator_allocated:755718 allocator_active:3808256 allocator_resident:3808256 total_system_memory:8053796864 total_system_memory_human:7.50G used_memory_lua:37888 used_memory_lua_human:37.00K used_memory_scripts:0 used_memory_scripts_human:0B number_of_cached_scripts:0 maxmemory:0 maxmemory_human:0B maxmemory_policy:noeviction allocator_frag_ratio:5.04 allocator_frag_bytes:3052538 allocator_rss_ratio:1.00 allocator_rss_bytes:0 rss_overhead_ratio:1.01 rss_overhead_bytes:37888 mem_fragmentation_ratio:5.09 mem_fragmentation_bytes:3090426 mem_not_counted_for_evict:0 mem_replication_backlog:0 mem_clients_slaves:0 mem_clients_normal:83538 mem_aof_buffer:0 mem_allocator:libc active_defrag_running:0 lazyfree_pending_objects:0

Persistence

loading:0 rdb_changes_since_last_save:2 rdb_bgsave_in_progress:0 rdb_last_save_time:1649865643 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:0 rdb_current_bgsave_time_sec:-1 rdb_last_cow_size:487424 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok aof_last_cow_size:0

Stats

total_connections_received:3015 total_commands_processed:231609196 instantaneous_ops_per_sec:1 total_net_input_bytes:31338644618 total_net_output_bytes:1584414740 instantaneous_input_kbps:0.17 instantaneous_output_kbps:0.01 rejected_connections:0 sync_full:0 sync_partial_ok:0 sync_partial_err:0 expired_keys:0 expired_stale_perc:0.00 expired_time_cap_reached_count:0 evicted_keys:0 keyspace_hits:5367534 keyspace_misses:7711769 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:116 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:0 active_defrag_misses:0 active_defrag_key_hits:0 active_defrag_key_misses:0

Replication

role:master connected_slaves:0 master_replid:a5f09e94d96c9a74e05086116c8f27b5e64e4671 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:0 second_repl_offset:-1 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0

CPU

used_cpu_sys:10506.235012 used_cpu_user:9990.253246 used_cpu_sys_children:75.834772 used_cpu_user_children:403.868009

Cluster

cluster_enabled:0

Keyspace

db0:keys=1243,expires=0,avg_ttl=0

Disk Usage:

df -h Filesystem Size Used Available Use% Mounted on overlay 98.3G 4.8G 93.1G 5% / tmpfs 64.0M 0 64.0M 0% /dev tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup /dev/nvme0n1p1 98.3G 4.8G 93.1G 5% /etc/resolv.conf /dev/nvme0n1p1 98.3G 4.8G 93.1G 5% /etc/hostname /dev/nvme0n1p1 98.3G 4.8G 93.1G 5% /etc/hosts shm 64.0M 0 64.0M 0% /dev/shm fs-006ac882.efs.us-east-1.amazonaws.com:/ 8.0E 0 8.0E 0% /var/lib/faktory tmpfs 3.8G 0 3.8G 0% /proc/acpi tmpfs 64.0M 0 64.0M 0% /proc/kcore tmpfs 64.0M 0 64.0M 0% /proc/keys tmpfs 64.0M 0 64.0M 0% /proc/latency_stats tmpfs 64.0M 0 64.0M 0% /proc/timer_list tmpfs 64.0M 0 64.0M 0% /proc/sched_debug tmpfs 3.8G 0 3.8G 0% /sys/firmware

— Reply to this email directly, view it on GitHub https://github.com/contribsys/faktory/issues/373#issuecomment-1098229909, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAWX2FGI6X7GBRP5QEU2TVE3WDDANCNFSM5BE6CN4Q . You are receiving this because you were mentioned.Message ID: @.***>

bploetz commented 2 years ago

@mperham not quickly, but we'll get that work planned.

bploetz commented 2 years ago

@mperham OK, just ran into this again after upgrading to the latest and greatest Faktory. Here's the relevant info:

Thread dump:

2022/07/04 19:23:00.140693 FULL PROCESS THREAD DUMP:
--
2022/07/04 19:23:00.140757 goroutine 1 [running]:
github.com/contribsys/faktory_worker_go.dumpThreads({0xe3d2e8, 0xc00000e048})
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:240 +0x4d
github.com/contribsys/faktory_worker_go.(*Manager).handleEvent(0xc0002ae1a0, {0xc98978, 0x15})
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:249 +0x105
github.com/contribsys/faktory_worker_go.(*Manager).Run(0xe1eea0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:187 +0x88
main.setupWorkers(0xc000238960, 0xc000230500, 0xc000230600, 0xc000230800, 0xc000230700, 0xc000230900, 0xc000230a00, 0xc000230b00, 0xc000230c00, {0xe4aa70, ...})
/opt/reward-service/workers/work.go:150 +0x150f
main.main()
/opt/reward-service/workers/work.go:75 +0x6f0
goroutine 18 [select]:
github.com/newrelic/go-agent/v3/newrelic.(*app).process(0xc0002d5200)
/go/pkg/mod/github.com/newrelic/go-agent/v3@v3.9.0/newrelic/internal_app.go:223 +0x1dc
created by github.com/newrelic/go-agent/v3/newrelic.newApp
/go/pkg/mod/github.com/newrelic/go-agent/v3@v3.9.0/newrelic/internal_app.go:431 +0x6e6
goroutine 1114 [select, 4362 minutes]:
net/http.(*persistConn).readLoop(0xc00007dd40)
/usr/local/go/src/net/http/transport.go:2207 +0xd8a
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1747 +0x1e05
goroutine 20 [select, 1 minutes]:
github.com/newrelic/go-agent/v3/newrelic.runSampler(0xc0002d5200, 0x0)
/go/pkg/mod/github.com/newrelic/go-agent/v3@v3.9.0/newrelic/internal_app.go:339 +0x245
created by github.com/newrelic/go-agent/v3/newrelic.newApp
/go/pkg/mod/github.com/newrelic/go-agent/v3@v3.9.0/newrelic/internal_app.go:434 +0x79d
goroutine 21 [select, 4376 minutes]:
database/sql.(*DB).connectionOpener(0xc0002af1e0, {0xe2e7d8, 0xc00023ae40})
/usr/local/go/src/database/sql/sql.go:1196 +0x93
created by database/sql.OpenDB
/usr/local/go/src/database/sql/sql.go:794 +0x188
goroutine 1937 [select, 4362 minutes]:
net/http.(*persistConn).writeLoop(0xc0007d3e60)
/usr/local/go/src/net/http/transport.go:2386 +0xfb
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1748 +0x1e65
goroutine 34 [semacquire, 49 minutes]:
internal/poll.runtime_Semacquire(0x6a3f5b)
/usr/local/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc00036a400, 0x35)
/usr/local/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).readLock(...)
/usr/local/go/src/internal/poll/fd_mutex.go:221
internal/poll.(*FD).Read(0xc00036a400, {0xc00031c04b, 0xfb5, 0xfb5})
/usr/local/go/src/internal/poll/fd_unix.go:144 +0x65
net.(*netFD).Read(0xc00036a400, {0xc00031c04b, 0xec6a2ef23e87, 0x195})
/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00000e090, {0xc00031c04b, 0xc0000011e0, 0x200000003})
/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).fill(0xc0004e4840)
/usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).ReadSlice(0xc0004e4840, 0x0)
/usr/local/go/src/bufio/bufio.go:360 +0x2f
bufio.(*Reader).collectFragments(0x8, 0x80)
/usr/local/go/src/bufio/bufio.go:435 +0x74
bufio.(*Reader).ReadBytes(0xc0004e4840, 0x10)
/usr/local/go/src/bufio/bufio.go:463 +0x1d
github.com/contribsys/faktory/client.readResponse(0xc00052aa90)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:630 +0x45
github.com/contribsys/faktory/client.ok(0xd0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:593 +0x1d
github.com/contribsys/faktory/client.(*Client).ok(0xc0004c8340, 0xc00044b040)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:527 +0x25
github.com/contribsys/faktory/client.(*Client).Push(0xc0004c8340, 0xe)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:323 +0x74
github.com/merryfield/reward-service/processors.enqueueUpdateProfileJob(0xc000794ec0, {0xc000c7e5d0, 0x24}, {0xe1d760, 0xc0004c8340})
/opt/reward-service/processors/receipt_processing_helpers.go:759 +0x248
github.com/merryfield/reward-service/processors.(*DigitalReceiptProcessor).Process(0xc00023e510, {0xe2e880, 0xc00065df80}, {0xc000794ec0, 0x1, 0x4})
/opt/reward-service/processors/digital_receipt.go:223 +0x2c6b
github.com/contribsys/faktory_worker_go.(*Manager).Register.func1({0xe2e880, 0xc00065df80}, 0xc00360)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:47 +0x37
github.com/contribsys/faktory_worker_go.dispatch({0x0, 0xc002345f10, 0xc00105af70}, {0xe2e880, 0xc00065df80}, 0x3, 0x40cef4)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/middleware.go:19 +0x118
github.com/contribsys/faktory_worker_go.processOne(0xc0002ae1a0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:139 +0x145
github.com/contribsys/faktory_worker_go.process(0xc0002ae1a0, 0xc0002d6fc0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:95 +0xc7
created by github.com/contribsys/faktory_worker_go.(*Manager).boot
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:172 +0x1df
goroutine 1015 [select]:
github.com/segmentio/kafka-go.(*connPool).discover(0xc0002db5c0, {0xe2e7d8, 0xc000794940}, 0xc000680d20)
/go/pkg/mod/github.com/segmentio/kafka-go@v0.4.8/transport.go:578 +0x4bd
created by github.com/segmentio/kafka-go.(*Transport).grabPool
/go/pkg/mod/github.com/segmentio/kafka-go@v0.4.8/transport.go:254 +0x5a7
goroutine 36 [semacquire, 80 minutes]:
internal/poll.runtime_Semacquire(0x6a3f5b)
/usr/local/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc00036a400, 0x35)
/usr/local/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).readLock(...)
/usr/local/go/src/internal/poll/fd_mutex.go:221
internal/poll.(*FD).Read(0xc00036a400, {0xc00031c017, 0xfe9, 0xfe9})
/usr/local/go/src/internal/poll/fd_unix.go:144 +0x65
net.(*netFD).Read(0xc00036a400, {0xc00031c017, 0x403a49, 0x195})
/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00000e090, {0xc00031c017, 0xc000001520, 0x200000003})
/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).fill(0xc0004e4840)
/usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).ReadSlice(0xc0004e4840, 0x0)
/usr/local/go/src/bufio/bufio.go:360 +0x2f
bufio.(*Reader).collectFragments(0x8, 0x80)
/usr/local/go/src/bufio/bufio.go:435 +0x74
bufio.(*Reader).ReadBytes(0xc0004e4840, 0x10)
/usr/local/go/src/bufio/bufio.go:463 +0x1d
github.com/contribsys/faktory/client.readResponse(0xc0002fcd00)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:630 +0x45
github.com/contribsys/faktory/client.ok(0xd0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:593 +0x1d
github.com/contribsys/faktory/client.(*Client).ok(0xc0004c8340, 0xc0009a5b80)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:527 +0x25
github.com/contribsys/faktory/client.(*Client).Push(0xc0004c8340, 0xe)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:323 +0x74
github.com/merryfield/reward-service/processors.enqueueUpdateProfileJob(0xc0009acfc0, {0xc001a36570, 0x24}, {0xe1d760, 0xc0004c8340})
/opt/reward-service/processors/receipt_processing_helpers.go:759 +0x248
github.com/merryfield/reward-service/processors.(*DigitalReceiptProcessor).Process(0xc00023e510, {0xe2e880, 0xc000d19470}, {0xc0009acfc0, 0x1, 0x4})
/opt/reward-service/processors/digital_receipt.go:223 +0x2c6b
github.com/contribsys/faktory_worker_go.(*Manager).Register.func1({0xe2e880, 0xc000d19470}, 0xc00360)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:47 +0x37
github.com/contribsys/faktory_worker_go.dispatch({0x0, 0xc00127df10, 0xc0016ce9a0}, {0xe2e880, 0xc000d19470}, 0x3, 0x40cef4)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/middleware.go:19 +0x118
github.com/contribsys/faktory_worker_go.processOne(0xc0002ae1a0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:139 +0x145
github.com/contribsys/faktory_worker_go.process(0xc0002ae1a0, 0xc0002d5200)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:95 +0xc7
created by github.com/contribsys/faktory_worker_go.(*Manager).boot
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:172 +0x1df
goroutine 32 [select]:
github.com/contribsys/faktory_worker_go.heartbeat(0xc0002ae1a0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:39 +0x105
created by github.com/contribsys/faktory_worker_go.(*Manager).boot
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:167 +0x7f
goroutine 35 [semacquire, 62 minutes]:
internal/poll.runtime_Semacquire(0x6a3f5b)
/usr/local/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc00036a400, 0x35)
/usr/local/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).readLock(...)
/usr/local/go/src/internal/poll/fd_mutex.go:221
internal/poll.(*FD).Read(0xc00036a400, {0xc00031c01f, 0xfe1, 0xfe1})
/usr/local/go/src/internal/poll/fd_unix.go:144 +0x65
net.(*netFD).Read(0xc00036a400, {0xc00031c01f, 0x403a49, 0x195})
/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00000e090, {0xc00031c01f, 0xc000001ba0, 0x200000003})
/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).fill(0xc0004e4840)
/usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).ReadSlice(0xc0004e4840, 0x0)
/usr/local/go/src/bufio/bufio.go:360 +0x2f
bufio.(*Reader).collectFragments(0x8, 0x80)
/usr/local/go/src/bufio/bufio.go:435 +0x74
bufio.(*Reader).ReadBytes(0xc0004e4840, 0x10)
/usr/local/go/src/bufio/bufio.go:463 +0x1d
github.com/contribsys/faktory/client.readResponse(0xc0002fc9c0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:630 +0x45
github.com/contribsys/faktory/client.ok(0xd0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:593 +0x1d
github.com/contribsys/faktory/client.(*Client).ok(0xc0004c8340, 0xc0007f5e00)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:527 +0x25
github.com/contribsys/faktory/client.(*Client).Push(0xc0004c8340, 0xe)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:323 +0x74
github.com/merryfield/reward-service/processors.enqueueUpdateProfileJob(0xc00023a180, {0xc001e9c420, 0x24}, {0xe1d760, 0xc0004c8340})
/opt/reward-service/processors/receipt_processing_helpers.go:759 +0x248
github.com/merryfield/reward-service/processors.(*DigitalReceiptProcessor).Process(0xc00023e510, {0xe2e880, 0xc00088e120}, {0xc00023a180, 0x1, 0x4})
/opt/reward-service/processors/digital_receipt.go:223 +0x2c6b
github.com/contribsys/faktory_worker_go.(*Manager).Register.func1({0xe2e880, 0xc00088e120}, 0xc00360)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:47 +0x37
github.com/contribsys/faktory_worker_go.dispatch({0x0, 0xc00212bf10, 0xc0010cec40}, {0xe2e880, 0xc00088e120}, 0x3, 0x40cef4)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/middleware.go:19 +0x118
github.com/contribsys/faktory_worker_go.processOne(0xc0002ae1a0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:139 +0x145
github.com/contribsys/faktory_worker_go.process(0xc0002ae1a0, 0xc0002d6fc0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:95 +0xc7
created by github.com/contribsys/faktory_worker_go.(*Manager).boot
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:172 +0x1df
goroutine 33 [semacquire, 81 minutes]:
internal/poll.runtime_Semacquire(0x6a3f5b)
/usr/local/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc00036a400, 0x35)
/usr/local/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).readLock(...)
/usr/local/go/src/internal/poll/fd_mutex.go:221
internal/poll.(*FD).Read(0xc00036a400, {0xc00031c00b, 0xff5, 0xff5})
/usr/local/go/src/internal/poll/fd_unix.go:144 +0x65
net.(*netFD).Read(0xc00036a400, {0xc00031c00b, 0x403a49, 0x195})
/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00000e090, {0xc00031c00b, 0xc000092000, 0x200000003})
/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).fill(0xc0004e4840)
/usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).ReadSlice(0xc0004e4840, 0x0)
/usr/local/go/src/bufio/bufio.go:360 +0x2f
bufio.(*Reader).collectFragments(0x8, 0x80)
/usr/local/go/src/bufio/bufio.go:435 +0x74
bufio.(*Reader).ReadBytes(0xc0004e4840, 0x10)
/usr/local/go/src/bufio/bufio.go:463 +0x1d
github.com/contribsys/faktory/client.readResponse(0xc001a14c30)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:630 +0x45
github.com/contribsys/faktory/client.ok(0xd0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:593 +0x1d
github.com/contribsys/faktory/client.(*Client).ok(0xc0004c8340, 0xc000b16320)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:527 +0x25
github.com/contribsys/faktory/client.(*Client).Push(0xc0004c8340, 0xe)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:323 +0x74
github.com/merryfield/reward-service/processors.enqueueUpdateProfileJob(0xc000204300, {0xc001ecec60, 0x24}, {0xe1d760, 0xc0004c8340})
/opt/reward-service/processors/receipt_processing_helpers.go:759 +0x248
github.com/merryfield/reward-service/processors.(*DigitalReceiptProcessor).Process(0xc00023e510, {0xe2e880, 0xc00065dd70}, {0xc000204300, 0x1, 0x4})
/opt/reward-service/processors/digital_receipt.go:223 +0x2c6b
github.com/contribsys/faktory_worker_go.(*Manager).Register.func1({0xe2e880, 0xc00065dd70}, 0xc00360)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:47 +0x37
github.com/contribsys/faktory_worker_go.dispatch({0x0, 0xc001f5ff10, 0xc00115c6d0}, {0xe2e880, 0xc00065dd70}, 0x3, 0x40cef4)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/middleware.go:19 +0x118
github.com/contribsys/faktory_worker_go.processOne(0xc0002ae1a0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:139 +0x145
github.com/contribsys/faktory_worker_go.process(0xc0002ae1a0, 0x1)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:95 +0xc7
created by github.com/contribsys/faktory_worker_go.(*Manager).boot
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:172 +0x1df
goroutine 965 [select, 4362 minutes]:
net/http.(*persistConn).readLoop(0xc000106120)
/usr/local/go/src/net/http/transport.go:2207 +0xd8a
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1747 +0x1e05
goroutine 37 [semacquire, 63 minutes]:
internal/poll.runtime_Semacquire(0x6a3f5b)
/usr/local/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc00036a400, 0x35)
/usr/local/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).readLock(...)
/usr/local/go/src/internal/poll/fd_mutex.go:221
internal/poll.(*FD).Read(0xc00036a400, {0xc00031c037, 0xfc9, 0xfc9})
/usr/local/go/src/internal/poll/fd_unix.go:144 +0x65
net.(*netFD).Read(0xc00036a400, {0xc00031c037, 0x403a49, 0x195})
/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00000e090, {0xc00031c037, 0xc000001d40, 0x200000003})
/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).fill(0xc0004e4840)
/usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).ReadSlice(0xc0004e4840, 0x0)
/usr/local/go/src/bufio/bufio.go:360 +0x2f
bufio.(*Reader).collectFragments(0x8, 0x80)
/usr/local/go/src/bufio/bufio.go:435 +0x74
bufio.(*Reader).ReadBytes(0xc0004e4840, 0x10)
/usr/local/go/src/bufio/bufio.go:463 +0x1d
github.com/contribsys/faktory/client.readResponse(0xc00052aea0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:630 +0x45
github.com/contribsys/faktory/client.ok(0xd0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:593 +0x1d
github.com/contribsys/faktory/client.(*Client).ok(0xc0004c8340, 0xc000a85b80)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:527 +0x25
github.com/contribsys/faktory/client.(*Client).Push(0xc0004c8340, 0xe)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:323 +0x74
github.com/merryfield/reward-service/processors.enqueueUpdateProfileJob(0xc000b3d200, {0xc0012f74d0, 0x24}, {0xe1d760, 0xc0004c8340})
/opt/reward-service/processors/receipt_processing_helpers.go:759 +0x248
github.com/merryfield/reward-service/processors.(*DigitalReceiptProcessor).Process(0xc00023e510, {0xe2e880, 0xc00095ab40}, {0xc000b3d200, 0x1, 0x4})
/opt/reward-service/processors/digital_receipt.go:223 +0x2c6b
github.com/contribsys/faktory_worker_go.(*Manager).Register.func1({0xe2e880, 0xc00095ab40}, 0xc00360)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:47 +0x37
github.com/contribsys/faktory_worker_go.dispatch({0x0, 0xc0016a9f10, 0xc0018268b0}, {0xe2e880, 0xc00095ab40}, 0x3, 0x40cef4)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/middleware.go:19 +0x118
github.com/contribsys/faktory_worker_go.processOne(0xc0002ae1a0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:139 +0x145
github.com/contribsys/faktory_worker_go.process(0xc0002ae1a0, 0x1c)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:95 +0xc7
created by github.com/contribsys/faktory_worker_go.(*Manager).boot
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:172 +0x1df
goroutine 38 [semacquire, 62 minutes]:
internal/poll.runtime_Semacquire(0x6a3f5b)
/usr/local/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc00036a400, 0x35)
/usr/local/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).readLock(...)
/usr/local/go/src/internal/poll/fd_mutex.go:221
internal/poll.(*FD).Read(0xc00036a400, {0xc00031c03f, 0xfc1, 0xfc1})
/usr/local/go/src/internal/poll/fd_unix.go:144 +0x65
net.(*netFD).Read(0xc00036a400, {0xc00031c03f, 0x403a49, 0x195})
/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00000e090, {0xc00031c03f, 0xc000143d40, 0x200000003})
/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).fill(0xc0004e4840)
/usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).ReadSlice(0xc0004e4840, 0x0)
/usr/local/go/src/bufio/bufio.go:360 +0x2f
bufio.(*Reader).collectFragments(0x8, 0x80)
/usr/local/go/src/bufio/bufio.go:435 +0x74
bufio.(*Reader).ReadBytes(0xc0004e4840, 0x10)
/usr/local/go/src/bufio/bufio.go:463 +0x1d
github.com/contribsys/faktory/client.readResponse(0xc00052b2b0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:630 +0x45
github.com/contribsys/faktory/client.ok(0xd0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:593 +0x1d
github.com/contribsys/faktory/client.(*Client).ok(0xc0004c8340, 0xc00044a6e0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:527 +0x25
github.com/contribsys/faktory/client.(*Client).Push(0xc0004c8340, 0xe)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:323 +0x74
github.com/merryfield/reward-service/processors.enqueueUpdateProfileJob(0xc00094f400, {0xc000e21aa0, 0x24}, {0xe1d760, 0xc0004c8340})
/opt/reward-service/processors/receipt_processing_helpers.go:759 +0x248
github.com/merryfield/reward-service/processors.(*DigitalReceiptProcessor).Process(0xc00023e510, {0xe2e880, 0xc0006e51a0}, {0xc00094f400, 0x1, 0x4})
/opt/reward-service/processors/digital_receipt.go:223 +0x2c6b
github.com/contribsys/faktory_worker_go.(*Manager).Register.func1({0xe2e880, 0xc0006e51a0}, 0xc00360)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:47 +0x37
github.com/contribsys/faktory_worker_go.dispatch({0x0, 0xc001003f10, 0xc001dc3990}, {0xe2e880, 0xc0006e51a0}, 0x3, 0x40cef4)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/middleware.go:19 +0x118
github.com/contribsys/faktory_worker_go.processOne(0xc0002ae1a0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:139 +0x145
github.com/contribsys/faktory_worker_go.process(0xc0002ae1a0, 0x0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:95 +0xc7
created by github.com/contribsys/faktory_worker_go.(*Manager).boot
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:172 +0x1df
goroutine 39 [semacquire, 73 minutes]:
internal/poll.runtime_Semacquire(0x6a3f5b)
/usr/local/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc00036a400, 0x35)
/usr/local/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).readLock(...)
/usr/local/go/src/internal/poll/fd_mutex.go:221
internal/poll.(*FD).Read(0xc00036a400, {0xc00031c01b, 0xfe5, 0xfe5})
/usr/local/go/src/internal/poll/fd_unix.go:144 +0x65
net.(*netFD).Read(0xc00036a400, {0xc00031c01b, 0xea8cb85b955a, 0x195})
/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00000e090, {0xc00031c01b, 0xc000198000, 0x200000003})
/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).fill(0xc0004e4840)
/usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).ReadSlice(0xc0004e4840, 0x50)
/usr/local/go/src/bufio/bufio.go:360 +0x2f
bufio.(*Reader).collectFragments(0x7f8ad5aff798, 0x0)
/usr/local/go/src/bufio/bufio.go:435 +0x74
bufio.(*Reader).ReadBytes(0xc0004e4840, 0x4)
/usr/local/go/src/bufio/bufio.go:463 +0x1d
github.com/contribsys/faktory/client.readResponse(0xc0002fc000)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:630 +0x45
github.com/contribsys/faktory/client.ok(0xd0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:593 +0x1d
github.com/contribsys/faktory/client.(*Client).ok(0xc0004c8340, 0xc0009a4fa0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:527 +0x25
github.com/contribsys/faktory/client.(*Client).Push(0xc0004c8340, 0xe)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:323 +0x74
github.com/merryfield/reward-service/processors.enqueueUpdateProfileJob(0xc0009ad940, {0xc0017d4090, 0x24}, {0xe1d760, 0xc0004c8340})
/opt/reward-service/processors/receipt_processing_helpers.go:759 +0x248
github.com/merryfield/reward-service/processors.(*DigitalReceiptProcessor).Process(0xc00023e510, {0xe2e880, 0xc0014c2de0}, {0xc0009ad940, 0x1, 0x4})
/opt/reward-service/processors/digital_receipt.go:223 +0x2c6b
github.com/contribsys/faktory_worker_go.(*Manager).Register.func1({0xe2e880, 0xc0014c2de0}, 0xc00360)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:47 +0x37
github.com/contribsys/faktory_worker_go.dispatch({0x0, 0xc001ac3f10, 0xc000f87800}, {0xe2e880, 0xc0014c2de0}, 0x3, 0x40cef4)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/middleware.go:19 +0x118
github.com/contribsys/faktory_worker_go.processOne(0xc0002ae1a0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:139 +0x145
github.com/contribsys/faktory_worker_go.process(0xc0002ae1a0, 0x0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:95 +0xc7
created by github.com/contribsys/faktory_worker_go.(*Manager).boot
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:172 +0x1df
goroutine 40 [semacquire, 57 minutes]:
internal/poll.runtime_Semacquire(0x6a3f5b)
/usr/local/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc00036a400, 0x35)
/usr/local/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).readLock(...)
/usr/local/go/src/internal/poll/fd_mutex.go:221
internal/poll.(*FD).Read(0xc00036a400, {0xc00031c047, 0xfb9, 0xfb9})
/usr/local/go/src/internal/poll/fd_unix.go:144 +0x65
net.(*netFD).Read(0xc00036a400, {0xc00031c047, 0xebfc0ed1c5af, 0x195})
/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00000e090, {0xc00031c047, 0xc0001981a0, 0x200000003})
/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).fill(0xc0004e4840)
/usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).ReadSlice(0xc0004e4840, 0x0)
/usr/local/go/src/bufio/bufio.go:360 +0x2f
bufio.(*Reader).collectFragments(0x7f8ad5884df8, 0x0)
/usr/local/go/src/bufio/bufio.go:435 +0x74
bufio.(*Reader).ReadBytes(0xc0004e4840, 0x4)
/usr/local/go/src/bufio/bufio.go:463 +0x1d
github.com/contribsys/faktory/client.readResponse(0xc00052a9c0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:630 +0x45
github.com/contribsys/faktory/client.ok(0xd0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:593 +0x1d
github.com/contribsys/faktory/client.(*Client).ok(0xc0004c8340, 0xc000b17180)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:527 +0x25
github.com/contribsys/faktory/client.(*Client).Push(0xc0004c8340, 0xe)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:323 +0x74
github.com/merryfield/reward-service/processors.enqueueUpdateProfileJob(0xc00021cb00, {0xc001aaac60, 0x24}, {0xe1d760, 0xc0004c8340})
/opt/reward-service/processors/receipt_processing_helpers.go:759 +0x248
github.com/merryfield/reward-service/processors.(*DigitalReceiptProcessor).Process(0xc00023e510, {0xe2e880, 0xc0003497a0}, {0xc00021cb00, 0x1, 0x4})
/opt/reward-service/processors/digital_receipt.go:223 +0x2c6b
github.com/contribsys/faktory_worker_go.(*Manager).Register.func1({0xe2e880, 0xc0003497a0}, 0xc00360)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:47 +0x37
github.com/contribsys/faktory_worker_go.dispatch({0x0, 0xc001987f10, 0xc001d0a570}, {0xe2e880, 0xc0003497a0}, 0x3, 0x40cef4)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/middleware.go:19 +0x118
github.com/contribsys/faktory_worker_go.processOne(0xc0002ae1a0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:139 +0x145
github.com/contribsys/faktory_worker_go.process(0xc0002ae1a0, 0x0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:95 +0xc7
created by github.com/contribsys/faktory_worker_go.(*Manager).boot
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:172 +0x1df
goroutine 41 [semacquire, 62 minutes]:
internal/poll.runtime_Semacquire(0x6a3f5b)
/usr/local/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc00036a400, 0x35)
/usr/local/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).readLock(...)
/usr/local/go/src/internal/poll/fd_mutex.go:221
internal/poll.(*FD).Read(0xc00036a400, {0xc00031c003, 0xffd, 0xffd})
/usr/local/go/src/internal/poll/fd_unix.go:144 +0x65
net.(*netFD).Read(0xc00036a400, {0xc00031c003, 0x403a49, 0x195})
/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00000e090, {0xc00031c003, 0xc000198680, 0x200000003})
/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).fill(0xc0004e4840)
/usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).ReadSlice(0xc0004e4840, 0x0)
/usr/local/go/src/bufio/bufio.go:360 +0x2f
bufio.(*Reader).collectFragments(0x8, 0x80)
/usr/local/go/src/bufio/bufio.go:435 +0x74
bufio.(*Reader).ReadBytes(0xc0004e4840, 0x10)
/usr/local/go/src/bufio/bufio.go:463 +0x1d
github.com/contribsys/faktory/client.readResponse(0xc0002fd1e0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:630 +0x45
github.com/contribsys/faktory/client.ok(0xd0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:593 +0x1d
github.com/contribsys/faktory/client.(*Client).ok(0xc0004c8340, 0xc00094b860)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:527 +0x25
github.com/contribsys/faktory/client.(*Client).Push(0xc0004c8340, 0xe)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:323 +0x74
github.com/merryfield/reward-service/processors.enqueueUpdateProfileJob(0xc000204480, {0xc001eced20, 0x24}, {0xe1d760, 0xc0004c8340})
/opt/reward-service/processors/receipt_processing_helpers.go:759 +0x248
github.com/merryfield/reward-service/processors.(*DigitalReceiptProcessor).Process(0xc00023e510, {0xe2e880, 0xc00065df80}, {0xc000204480, 0x1, 0x4})
/opt/reward-service/processors/digital_receipt.go:223 +0x2c6b
github.com/contribsys/faktory_worker_go.(*Manager).Register.func1({0xe2e880, 0xc00065df80}, 0xc00360)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:47 +0x37
github.com/contribsys/faktory_worker_go.dispatch({0x0, 0xc0013fdf10, 0xc00115c730}, {0xe2e880, 0xc00065df80}, 0x3, 0x40cef4)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/middleware.go:19 +0x118
github.com/contribsys/faktory_worker_go.processOne(0xc0002ae1a0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:139 +0x145
github.com/contribsys/faktory_worker_go.process(0xc0002ae1a0, 0x0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:95 +0xc7
created by github.com/contribsys/faktory_worker_go.(*Manager).boot
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:172 +0x1df
goroutine 42 [semacquire, 80 minutes]:
internal/poll.runtime_Semacquire(0x6a3f5b)
/usr/local/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc00036a400, 0x35)
/usr/local/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).readLock(...)
/usr/local/go/src/internal/poll/fd_mutex.go:221
internal/poll.(*FD).Read(0xc00036a400, {0xc00031c02b, 0xfd5, 0xfd5})
/usr/local/go/src/internal/poll/fd_unix.go:144 +0x65
net.(*netFD).Read(0xc00036a400, {0xc00031c02b, 0xeaaf563774a3, 0x195})
/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00000e090, {0xc00031c02b, 0xc000198820, 0x200000003})
/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).fill(0xc0004e4840)
/usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).ReadSlice(0xc0004e4840, 0x0)
/usr/local/go/src/bufio/bufio.go:360 +0x2f
bufio.(*Reader).collectFragments(0x7f8afc7abf60, 0x0)
/usr/local/go/src/bufio/bufio.go:435 +0x74
bufio.(*Reader).ReadBytes(0xc0004e4840, 0x4)
/usr/local/go/src/bufio/bufio.go:463 +0x1d
github.com/contribsys/faktory/client.readResponse(0xc00052a9c0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:630 +0x45
github.com/contribsys/faktory/client.ok(0xd0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:593 +0x1d
github.com/contribsys/faktory/client.(*Client).ok(0xc0004c8340, 0xc0007f5c20)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:527 +0x25
github.com/contribsys/faktory/client.(*Client).Push(0xc0004c8340, 0xe)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:323 +0x74
github.com/merryfield/reward-service/processors.enqueueUpdateProfileJob(0xc000205d80, {0xc00105e9c0, 0x24}, {0xe1d760, 0xc0004c8340})
/opt/reward-service/processors/receipt_processing_helpers.go:759 +0x248
github.com/merryfield/reward-service/processors.(*DigitalReceiptProcessor).Process(0xc00023e510, {0xe2e880, 0xc0009f6690}, {0xc000205d80, 0x1, 0x4})
/opt/reward-service/processors/digital_receipt.go:223 +0x2c6b
github.com/contribsys/faktory_worker_go.(*Manager).Register.func1({0xe2e880, 0xc0009f6690}, 0xc00360)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:47 +0x37
github.com/contribsys/faktory_worker_go.dispatch({0x0, 0xc001bd5f10, 0xc0004dc3d0}, {0xe2e880, 0xc0009f6690}, 0x3, 0x40cef4)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/middleware.go:19 +0x118
github.com/contribsys/faktory_worker_go.processOne(0xc0002ae1a0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:139 +0x145
github.com/contribsys/faktory_worker_go.process(0xc0002ae1a0, 0x0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:95 +0xc7
created by github.com/contribsys/faktory_worker_go.(*Manager).boot
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:172 +0x1df
goroutine 43 [IO wait, 49 minutes]:
internal/poll.runtime_pollWait(0x7f8ad5b3a948, 0x72)
/usr/local/go/src/runtime/netpoll.go:303 +0x85
internal/poll.(*pollDesc).wait(0xc00036a400, 0xc00031c050, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00036a400, {0xc00031c050, 0xfb0, 0xfb0})
/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00036a400, {0xc00031c050, 0x200000003, 0xc0001989c0})
/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00000e090, {0xc00031c050, 0x4c3da5, 0x5})
/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).fill(0xc0004e4840)
/usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).ReadSlice(0xc0004e4840, 0xa8)
/usr/local/go/src/bufio/bufio.go:360 +0x2f
bufio.(*Reader).collectFragments(0xc00031c000, 0x80)
/usr/local/go/src/bufio/bufio.go:435 +0x74
bufio.(*Reader).ReadBytes(0xc0004e4840, 0x10)
/usr/local/go/src/bufio/bufio.go:463 +0x1d
github.com/contribsys/faktory/client.readResponse(0xc0002fc340)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:630 +0x45
github.com/contribsys/faktory/client.ok(0xd0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:593 +0x1d
github.com/contribsys/faktory/client.(*Client).ok(0xc0004c8340, 0xc0009a5d60)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:527 +0x25
github.com/contribsys/faktory/client.(*Client).Push(0xc0004c8340, 0xe)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:323 +0x74
github.com/merryfield/reward-service/processors.enqueueUpdateProfileJob(0xc0009ac740, {0xc001e9ccc0, 0x24}, {0xe1d760, 0xc0004c8340})
/opt/reward-service/processors/receipt_processing_helpers.go:759 +0x248
github.com/merryfield/reward-service/processors.(*DigitalReceiptProcessor).Process(0xc00023e510, {0xe2e880, 0xc000bf0c60}, {0xc0009ac740, 0x1, 0x4})
/opt/reward-service/processors/digital_receipt.go:223 +0x2c6b
github.com/contribsys/faktory_worker_go.(*Manager).Register.func1({0xe2e880, 0xc000bf0c60}, 0xc00360)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:47 +0x37
github.com/contribsys/faktory_worker_go.dispatch({0x0, 0xc001ce1f10, 0xc000e930f0}, {0xe2e880, 0xc000bf0c60}, 0x3, 0x40cef4)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/middleware.go:19 +0x118
github.com/contribsys/faktory_worker_go.processOne(0xc0002ae1a0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:139 +0x145
github.com/contribsys/faktory_worker_go.process(0xc0002ae1a0, 0x0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:95 +0xc7
created by github.com/contribsys/faktory_worker_go.(*Manager).boot
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:172 +0x1df
goroutine 44 [semacquire, 59 minutes]:
internal/poll.runtime_Semacquire(0x6a3f5b)
/usr/local/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc00036a400, 0x35)
/usr/local/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).readLock(...)
/usr/local/go/src/internal/poll/fd_mutex.go:221
internal/poll.(*FD).Read(0xc00036a400, {0xc00031c023, 0xfdd, 0xfdd})
/usr/local/go/src/internal/poll/fd_unix.go:144 +0x65
net.(*netFD).Read(0xc00036a400, {0xc00031c023, 0x403a49, 0x195})
/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00000e090, {0xc00031c023, 0xc000198b60, 0x200000003})
/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).fill(0xc0004e4840)
/usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).ReadSlice(0xc0004e4840, 0x0)
/usr/local/go/src/bufio/bufio.go:360 +0x2f
bufio.(*Reader).collectFragments(0x7f8ad5affe80, 0x0)
/usr/local/go/src/bufio/bufio.go:435 +0x74
bufio.(*Reader).ReadBytes(0xc0004e4840, 0x4)
/usr/local/go/src/bufio/bufio.go:463 +0x1d
github.com/contribsys/faktory/client.readResponse(0xc0002fcd00)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:630 +0x45
github.com/contribsys/faktory/client.ok(0xd0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:593 +0x1d
github.com/contribsys/faktory/client.(*Client).ok(0xc0004c8340, 0xc0002bd180)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:527 +0x25
github.com/contribsys/faktory/client.(*Client).Push(0xc0004c8340, 0xe)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:323 +0x74
github.com/merryfield/reward-service/processors.enqueueUpdateProfileJob(0xc00023a040, {0xc001e9c360, 0x24}, {0xe1d760, 0xc0004c8340})
/opt/reward-service/processors/receipt_processing_helpers.go:759 +0x248
github.com/merryfield/reward-service/processors.(*DigitalReceiptProcessor).Process(0xc00023e510, {0xe2e880, 0xc001865e90}, {0xc00023a040, 0x1, 0x4})
/opt/reward-service/processors/digital_receipt.go:223 +0x2c6b
github.com/contribsys/faktory_worker_go.(*Manager).Register.func1({0xe2e880, 0xc001865e90}, 0xc00360)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:47 +0x37
github.com/contribsys/faktory_worker_go.dispatch({0x0, 0xc00127ff10, 0xc0010cebf0}, {0xe2e880, 0xc001865e90}, 0x3, 0x40cef4)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/middleware.go:19 +0x118
github.com/contribsys/faktory_worker_go.processOne(0xc0002ae1a0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:139 +0x145
github.com/contribsys/faktory_worker_go.process(0xc0002ae1a0, 0x0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:95 +0xc7
created by github.com/contribsys/faktory_worker_go.(*Manager).boot
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:172 +0x1df
goroutine 45 [semacquire, 73 minutes]:
internal/poll.runtime_Semacquire(0x6a3f5b)
/usr/local/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc00036a400, 0x35)
/usr/local/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).readLock(...)
/usr/local/go/src/internal/poll/fd_mutex.go:221
internal/poll.(*FD).Read(0xc00036a400, {0xc00031c033, 0xfcd, 0xfcd})
/usr/local/go/src/internal/poll/fd_unix.go:144 +0x65
net.(*netFD).Read(0xc00036a400, {0xc00031c033, 0x403a49, 0x195})
/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00000e090, {0xc00031c033, 0xc000198d00, 0x200000003})
/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).fill(0xc0004e4840)
/usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).ReadSlice(0xc0004e4840, 0x0)
/usr/local/go/src/bufio/bufio.go:360 +0x2f
bufio.(*Reader).collectFragments(0x8, 0x80)
/usr/local/go/src/bufio/bufio.go:435 +0x74
bufio.(*Reader).ReadBytes(0xc0004e4840, 0x10)
/usr/local/go/src/bufio/bufio.go:463 +0x1d
github.com/contribsys/faktory/client.readResponse(0xc00052a9c0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:630 +0x45
github.com/contribsys/faktory/client.ok(0xd0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:593 +0x1d
github.com/contribsys/faktory/client.(*Client).ok(0xc0004c8340, 0xc00094aaa0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:527 +0x25
github.com/contribsys/faktory/client.(*Client).Push(0xc0004c8340, 0xe)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:323 +0x74
github.com/merryfield/reward-service/processors.enqueueUpdateProfileJob(0xc00023ad80, {0xc0017d5bf0, 0x24}, {0xe1d760, 0xc0004c8340})
/opt/reward-service/processors/receipt_processing_helpers.go:759 +0x248
github.com/merryfield/reward-service/processors.(*DigitalReceiptProcessor).Process(0xc00023e510, {0xe2e880, 0xc001d41920}, {0xc00023ad80, 0x1, 0x4})
/opt/reward-service/processors/digital_receipt.go:223 +0x2c6b
github.com/contribsys/faktory_worker_go.(*Manager).Register.func1({0xe2e880, 0xc001d41920}, 0xc00360)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:47 +0x37
github.com/contribsys/faktory_worker_go.dispatch({0x0, 0xc000441f10, 0xc000f377c0}, {0xe2e880, 0xc001d41920}, 0x3, 0x40cef4)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/middleware.go:19 +0x118
github.com/contribsys/faktory_worker_go.processOne(0xc0002ae1a0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:139 +0x145
github.com/contribsys/faktory_worker_go.process(0xc0002ae1a0, 0x0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:95 +0xc7
created by github.com/contribsys/faktory_worker_go.(*Manager).boot
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:172 +0x1df
goroutine 46 [semacquire, 73 minutes]:
internal/poll.runtime_Semacquire(0x6a3f5b)
/usr/local/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc00036a400, 0x35)
/usr/local/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).readLock(...)
/usr/local/go/src/internal/poll/fd_mutex.go:221
internal/poll.(*FD).Read(0xc00036a400, {0xc00031c007, 0xff9, 0xff9})
/usr/local/go/src/internal/poll/fd_unix.go:144 +0x65
net.(*netFD).Read(0xc00036a400, {0xc00031c007, 0x403a49, 0x195})
/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00000e090, {0xc00031c007, 0xc000198ea0, 0x200000003})
/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).fill(0xc0004e4840)
/usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).ReadSlice(0xc0004e4840, 0x0)
/usr/local/go/src/bufio/bufio.go:360 +0x2f
bufio.(*Reader).collectFragments(0x8, 0x80)
/usr/local/go/src/bufio/bufio.go:435 +0x74
bufio.(*Reader).ReadBytes(0xc0004e4840, 0x10)
/usr/local/go/src/bufio/bufio.go:463 +0x1d
github.com/contribsys/faktory/client.readResponse(0xc0002fd2b0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:630 +0x45
github.com/contribsys/faktory/client.ok(0xd0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:593 +0x1d
github.com/contribsys/faktory/client.(*Client).ok(0xc0004c8340, 0xc00073a320)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:527 +0x25
github.com/contribsys/faktory/client.(*Client).Push(0xc0004c8340, 0xe)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:323 +0x74
github.com/merryfield/reward-service/processors.enqueueUpdateProfileJob(0xc000204600, {0xc001ecee10, 0x24}, {0xe1d760, 0xc0004c8340})
/opt/reward-service/processors/receipt_processing_helpers.go:759 +0x248
github.com/merryfield/reward-service/processors.(*DigitalReceiptProcessor).Process(0xc00023e510, {0xe2e880, 0xc001638240}, {0xc000204600, 0x1, 0x4})
/opt/reward-service/processors/digital_receipt.go:223 +0x2c6b
github.com/contribsys/faktory_worker_go.(*Manager).Register.func1({0xe2e880, 0xc001638240}, 0xc00360)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:47 +0x37
github.com/contribsys/faktory_worker_go.dispatch({0x0, 0xc001ac3f10, 0xc00115c790}, {0xe2e880, 0xc001638240}, 0x3, 0x40cef4)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/middleware.go:19 +0x118
github.com/contribsys/faktory_worker_go.processOne(0xc0002ae1a0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:139 +0x145
github.com/contribsys/faktory_worker_go.process(0xc0002ae1a0, 0x0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:95 +0xc7
created by github.com/contribsys/faktory_worker_go.(*Manager).boot
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:172 +0x1df
goroutine 47 [semacquire, 73 minutes]:
internal/poll.runtime_Semacquire(0x6a3f5b)
/usr/local/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc00036a400, 0x35)
/usr/local/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).readLock(...)
/usr/local/go/src/internal/poll/fd_mutex.go:221
internal/poll.(*FD).Read(0xc00036a400, {0xc00031c02f, 0xfd1, 0xfd1})
/usr/local/go/src/internal/poll/fd_unix.go:144 +0x65
net.(*netFD).Read(0xc00036a400, {0xc00031c02f, 0x403a49, 0x195})
/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00000e090, {0xc00031c02f, 0xc000199040, 0x200000003})
/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).fill(0xc0004e4840)
/usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).ReadSlice(0xc0004e4840, 0x0)
/usr/local/go/src/bufio/bufio.go:360 +0x2f
bufio.(*Reader).collectFragments(0x7f8ad5884df8, 0x0)
/usr/local/go/src/bufio/bufio.go:435 +0x74
bufio.(*Reader).ReadBytes(0xc0004e4840, 0x4)
/usr/local/go/src/bufio/bufio.go:463 +0x1d
github.com/contribsys/faktory/client.readResponse(0xc0002fc680)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:630 +0x45
github.com/contribsys/faktory/client.ok(0xd0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:593 +0x1d
github.com/contribsys/faktory/client.(*Client).ok(0xc0004c8340, 0xc000b172c0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:527 +0x25
github.com/contribsys/faktory/client.(*Client).Push(0xc0004c8340, 0xe)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:323 +0x74
github.com/merryfield/reward-service/processors.enqueueUpdateProfileJob(0xc0004c8bc0, {0xc001f6fd10, 0x24}, {0xe1d760, 0xc0004c8340})
/opt/reward-service/processors/receipt_processing_helpers.go:759 +0x248
github.com/merryfield/reward-service/processors.(*DigitalReceiptProcessor).Process(0xc00023e510, {0xe2e880, 0xc001891200}, {0xc0004c8bc0, 0x1, 0x4})
/opt/reward-service/processors/digital_receipt.go:223 +0x2c6b
github.com/contribsys/faktory_worker_go.(*Manager).Register.func1({0xe2e880, 0xc001891200}, 0xc00360)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:47 +0x37
github.com/contribsys/faktory_worker_go.dispatch({0x0, 0xc0000b5f10, 0xc000b6f730}, {0xe2e880, 0xc001891200}, 0x3, 0x40cef4)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/middleware.go:19 +0x118
github.com/contribsys/faktory_worker_go.processOne(0xc0002ae1a0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:139 +0x145
github.com/contribsys/faktory_worker_go.process(0xc0002ae1a0, 0x0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:95 +0xc7
created by github.com/contribsys/faktory_worker_go.(*Manager).boot
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:172 +0x1df
goroutine 48 [semacquire, 62 minutes]:
internal/poll.runtime_Semacquire(0x6a3f5b)
/usr/local/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc00036a400, 0x35)
/usr/local/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).readLock(...)
/usr/local/go/src/internal/poll/fd_mutex.go:221
internal/poll.(*FD).Read(0xc00036a400, {0xc00031c03b, 0xfc5, 0xfc5})
/usr/local/go/src/internal/poll/fd_unix.go:144 +0x65
net.(*netFD).Read(0xc00036a400, {0xc00031c03b, 0x403a49, 0x195})
/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00000e090, {0xc00031c03b, 0xc0001991e0, 0x200000003})
/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).fill(0xc0004e4840)
/usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).ReadSlice(0xc0004e4840, 0x0)
/usr/local/go/src/bufio/bufio.go:360 +0x2f
bufio.(*Reader).collectFragments(0x7f8ad58ae7b0, 0x0)
/usr/local/go/src/bufio/bufio.go:435 +0x74
bufio.(*Reader).ReadBytes(0xc0004e4840, 0x4)
/usr/local/go/src/bufio/bufio.go:463 +0x1d
github.com/contribsys/faktory/client.readResponse(0xc0002fc8f0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:630 +0x45
github.com/contribsys/faktory/client.ok(0xd0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:593 +0x1d
github.com/contribsys/faktory/client.(*Client).ok(0xc0004c8340, 0xc0009a4b40)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:527 +0x25
github.com/contribsys/faktory/client.(*Client).Push(0xc0004c8340, 0xe)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:323 +0x74
github.com/merryfield/reward-service/processors.enqueueUpdateProfileJob(0xc00094f3c0, {0xc000e21950, 0x24}, {0xe1d760, 0xc0004c8340})
/opt/reward-service/processors/receipt_processing_helpers.go:759 +0x248
github.com/merryfield/reward-service/processors.(*DigitalReceiptProcessor).Process(0xc00023e510, {0xe2e880, 0xc0006e4fc0}, {0xc00094f3c0, 0x1, 0x4})
/opt/reward-service/processors/digital_receipt.go:223 +0x2c6b
github.com/contribsys/faktory_worker_go.(*Manager).Register.func1({0xe2e880, 0xc0006e4fc0}, 0xc00360)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:47 +0x37
github.com/contribsys/faktory_worker_go.dispatch({0x0, 0xc000441f10, 0xc001dc3940}, {0xe2e880, 0xc0006e4fc0}, 0x3, 0x40cef4)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/middleware.go:19 +0x118
github.com/contribsys/faktory_worker_go.processOne(0xc0002ae1a0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:139 +0x145
github.com/contribsys/faktory_worker_go.process(0xc0002ae1a0, 0x0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:95 +0xc7
created by github.com/contribsys/faktory_worker_go.(*Manager).boot
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:172 +0x1df
goroutine 49 [semacquire, 63 minutes]:
internal/poll.runtime_Semacquire(0x6a3f5b)
/usr/local/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc00036a400, 0x35)
/usr/local/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).readLock(...)
/usr/local/go/src/internal/poll/fd_mutex.go:221
internal/poll.(*FD).Read(0xc00036a400, {0xc00031c00f, 0xff1, 0xff1})
/usr/local/go/src/internal/poll/fd_unix.go:144 +0x65
net.(*netFD).Read(0xc00036a400, {0xc00031c00f, 0x403a49, 0x195})
/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00000e090, {0xc00031c00f, 0xc000199380, 0x200000003})
/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).fill(0xc0004e4840)
/usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).ReadSlice(0xc0004e4840, 0x50)
/usr/local/go/src/bufio/bufio.go:360 +0x2f
bufio.(*Reader).collectFragments(0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:435 +0x74
bufio.(*Reader).ReadBytes(0xc0004e4840, 0x10)
/usr/local/go/src/bufio/bufio.go:463 +0x1d
github.com/contribsys/faktory/client.readResponse(0xc0002fc000)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:630 +0x45
github.com/contribsys/faktory/client.ok(0xd0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:593 +0x1d
github.com/contribsys/faktory/client.(*Client).ok(0xc0004c8340, 0xc0009a41e0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:527 +0x25
github.com/contribsys/faktory/client.(*Client).Push(0xc0004c8340, 0xe)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:323 +0x74
github.com/merryfield/reward-service/processors.enqueueUpdateProfileJob(0xc00094fb00, {0xc000e2a750, 0x24}, {0xe1d760, 0xc0004c8340})
/opt/reward-service/processors/receipt_processing_helpers.go:759 +0x248
github.com/merryfield/reward-service/processors.(*DigitalReceiptProcessor).Process(0xc00023e510, {0xe2e880, 0xc002028510}, {0xc00094fb00, 0x1, 0x4})
/opt/reward-service/processors/digital_receipt.go:223 +0x2c6b
github.com/contribsys/faktory_worker_go.(*Manager).Register.func1({0xe2e880, 0xc002028510}, 0xc00360)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:47 +0x37
github.com/contribsys/faktory_worker_go.dispatch({0x0, 0xc001ce1f10, 0xc0005d8ab0}, {0xe2e880, 0xc002028510}, 0x3, 0x1)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/middleware.go:19 +0x118
github.com/contribsys/faktory_worker_go.processOne(0xc0002ae1a0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:139 +0x145
github.com/contribsys/faktory_worker_go.process(0xc0002ae1a0, 0x0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:95 +0xc7
created by github.com/contribsys/faktory_worker_go.(*Manager).boot
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:172 +0x1df
goroutine 50 [semacquire, 49 minutes]:
internal/poll.runtime_Semacquire(0x6a3f5b)
/usr/local/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc00036a400, 0x35)
/usr/local/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).readLock(...)
/usr/local/go/src/internal/poll/fd_mutex.go:221
internal/poll.(*FD).Read(0xc00036a400, {0xc00031c027, 0xfd9, 0xfd9})
/usr/local/go/src/internal/poll/fd_unix.go:144 +0x65
net.(*netFD).Read(0xc00036a400, {0xc00031c027, 0xeaaa99ca78b9, 0x195})
/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00000e090, {0xc00031c027, 0xc000199520, 0x200000003})
/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).fill(0xc0004e4840)
/usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).ReadSlice(0xc0004e4840, 0x0)
/usr/local/go/src/bufio/bufio.go:360 +0x2f
bufio.(*Reader).collectFragments(0x7f8ad5890da8, 0x0)
/usr/local/go/src/bufio/bufio.go:435 +0x74
bufio.(*Reader).ReadBytes(0xc0004e4840, 0x4)
/usr/local/go/src/bufio/bufio.go:463 +0x1d
github.com/contribsys/faktory/client.readResponse(0xc00052ad00)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:630 +0x45
github.com/contribsys/faktory/client.ok(0xd0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:593 +0x1d
github.com/contribsys/faktory/client.(*Client).ok(0xc0004c8340, 0xc0007f4000)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:527 +0x25
github.com/contribsys/faktory/client.(*Client).Push(0xc0004c8340, 0xe)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:323 +0x74
github.com/merryfield/reward-service/processors.enqueueUpdateProfileJob(0xc00022a5c0, {0xc001e9d3b0, 0x24}, {0xe1d760, 0xc0004c8340})
/opt/reward-service/processors/receipt_processing_helpers.go:759 +0x248
github.com/merryfield/reward-service/processors.(*DigitalReceiptProcessor).Process(0xc00023e510, {0xe2e880, 0xc000c6b9b0}, {0xc00022a5c0, 0x1, 0x4})
/opt/reward-service/processors/digital_receipt.go:223 +0x2c6b
github.com/contribsys/faktory_worker_go.(*Manager).Register.func1({0xe2e880, 0xc000c6b9b0}, 0xc00360)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:47 +0x37
github.com/contribsys/faktory_worker_go.dispatch({0x0, 0xc00114ff10, 0xc000f87d80}, {0xe2e880, 0xc000c6b9b0}, 0x3, 0x40cef4)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/middleware.go:19 +0x118
github.com/contribsys/faktory_worker_go.processOne(0xc0002ae1a0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:139 +0x145
github.com/contribsys/faktory_worker_go.process(0xc0002ae1a0, 0x0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:95 +0xc7
created by github.com/contribsys/faktory_worker_go.(*Manager).boot
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:172 +0x1df
goroutine 51 [semacquire, 59 minutes]:
internal/poll.runtime_Semacquire(0x6a3f5b)
/usr/local/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc00036a400, 0x35)
/usr/local/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).readLock(...)
/usr/local/go/src/internal/poll/fd_mutex.go:221
internal/poll.(*FD).Read(0xc00036a400, {0xc00031c043, 0xfbd, 0xfbd})
/usr/local/go/src/internal/poll/fd_unix.go:144 +0x65
net.(*netFD).Read(0xc00036a400, {0xc00031c043, 0x403a49, 0x195})
/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00000e090, {0xc00031c043, 0xc0001996c0, 0x200000003})
/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).fill(0xc0004e4840)
/usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).ReadSlice(0xc0004e4840, 0x50)
/usr/local/go/src/bufio/bufio.go:360 +0x2f
bufio.(*Reader).collectFragments(0x7f8ad5882ae8, 0x0)
/usr/local/go/src/bufio/bufio.go:435 +0x74
bufio.(*Reader).ReadBytes(0xc0004e4840, 0x4)
/usr/local/go/src/bufio/bufio.go:463 +0x1d
github.com/contribsys/faktory/client.readResponse(0xc0002fc000)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:630 +0x45
github.com/contribsys/faktory/client.ok(0xd0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:593 +0x1d
github.com/contribsys/faktory/client.(*Client).ok(0xc0004c8340, 0xc00073ae60)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:527 +0x25
github.com/contribsys/faktory/client.(*Client).Push(0xc0004c8340, 0xe)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:323 +0x74
github.com/merryfield/reward-service/processors.enqueueUpdateProfileJob(0xc0009ac980, {0xc001aab110, 0x24}, {0xe1d760, 0xc0004c8340})
/opt/reward-service/processors/receipt_processing_helpers.go:759 +0x248
github.com/merryfield/reward-service/processors.(*DigitalReceiptProcessor).Process(0xc00023e510, {0xe2e880, 0xc000a298c0}, {0xc0009ac980, 0x1, 0x4})
/opt/reward-service/processors/digital_receipt.go:223 +0x2c6b
github.com/contribsys/faktory_worker_go.(*Manager).Register.func1({0xe2e880, 0xc000a298c0}, 0xc00360)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:47 +0x37
github.com/contribsys/faktory_worker_go.dispatch({0x0, 0xc000441f10, 0xc001cbddd0}, {0xe2e880, 0xc000a298c0}, 0x3, 0x40cef4)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/middleware.go:19 +0x118
github.com/contribsys/faktory_worker_go.processOne(0xc0002ae1a0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:139 +0x145
github.com/contribsys/faktory_worker_go.process(0xc0002ae1a0, 0x0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:95 +0xc7
created by github.com/contribsys/faktory_worker_go.(*Manager).boot
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:172 +0x1df
goroutine 52 [semacquire, 57 minutes]:
internal/poll.runtime_Semacquire(0x6a3f5b)
/usr/local/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc00036a400, 0x35)
/usr/local/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).readLock(...)
/usr/local/go/src/internal/poll/fd_mutex.go:221
internal/poll.(*FD).Read(0xc00036a400, {0xc00031c013, 0xfed, 0xfed})
/usr/local/go/src/internal/poll/fd_unix.go:144 +0x65
net.(*netFD).Read(0xc00036a400, {0xc00031c013, 0x403a49, 0x195})
/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00000e090, {0xc00031c013, 0xc000199860, 0x200000003})
/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).fill(0xc0004e4840)
/usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).ReadSlice(0xc0004e4840, 0x0)
/usr/local/go/src/bufio/bufio.go:360 +0x2f
bufio.(*Reader).collectFragments(0x8, 0x80)
/usr/local/go/src/bufio/bufio.go:435 +0x74
bufio.(*Reader).ReadBytes(0xc0004e4840, 0x10)
/usr/local/go/src/bufio/bufio.go:463 +0x1d
github.com/contribsys/faktory/client.readResponse(0xc00052a9c0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:630 +0x45
github.com/contribsys/faktory/client.ok(0xd0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:593 +0x1d
github.com/contribsys/faktory/client.(*Client).ok(0xc0004c8340, 0xc00044bea0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:527 +0x25
github.com/contribsys/faktory/client.(*Client).Push(0xc0004c8340, 0xe)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:323 +0x74
github.com/merryfield/reward-service/processors.enqueueUpdateProfileJob(0xc00021c940, {0xc001dc11d0, 0x24}, {0xe1d760, 0xc0004c8340})
/opt/reward-service/processors/receipt_processing_helpers.go:759 +0x248
github.com/merryfield/reward-service/processors.(*DigitalReceiptProcessor).Process(0xc00023e510, {0xe2e880, 0xc00124c810}, {0xc00021c940, 0x1, 0x4})
/opt/reward-service/processors/digital_receipt.go:223 +0x2c6b
github.com/contribsys/faktory_worker_go.(*Manager).Register.func1({0xe2e880, 0xc00124c810}, 0xc00360)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:47 +0x37
github.com/contribsys/faktory_worker_go.dispatch({0x0, 0xc001ac3f10, 0xc0015720e0}, {0xe2e880, 0xc00124c810}, 0x3, 0x40cef4)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/middleware.go:19 +0x118
github.com/contribsys/faktory_worker_go.processOne(0xc0002ae1a0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:139 +0x145
github.com/contribsys/faktory_worker_go.process(0xc0002ae1a0, 0x0)
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/runner.go:95 +0xc7
created by github.com/contribsys/faktory_worker_go.(*Manager).boot
/go/pkg/mod/github.com/contribsys/faktory_worker_go@v1.6.0/manager.go:172 +0x1df
goroutine 54 [runnable]:
os/signal.process({0xe29a20, 0x12ce4e8})
/usr/local/go/src/os/signal/signal.go:241 +0x14b
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:24 +0x34
created by os/signal.Notify.func1.1
/usr/local/go/src/os/signal/signal.go:151 +0x2c
goroutine 967 [select, 4362 minutes]:
net/http.(*persistConn).readLoop(0xc0001065a0)
/usr/local/go/src/net/http/transport.go:2207 +0xd8a
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1747 +0x1e05
goroutine 2289 [select, 4355 minutes]:
net/http.(*persistConn).writeLoop(0xc0004297a0)
/usr/local/go/src/net/http/transport.go:2386 +0xfb
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1748 +0x1e65
goroutine 286 [select]:
github.com/segmentio/kafka-go.(*connPool).discover(0xc0002da480, {0xe2e7d8, 0xc0000a7500}, 0xc000988a80)
/go/pkg/mod/github.com/segmentio/kafka-go@v0.4.8/transport.go:578 +0x4bd
created by github.com/segmentio/kafka-go.(*Transport).grabPool
/go/pkg/mod/github.com/segmentio/kafka-go@v0.4.8/transport.go:254 +0x5a7
goroutine 117717 [select, 1673 minutes]:
net/http.(*persistConn).writeLoop(0xc0011e8480)
/usr/local/go/src/net/http/transport.go:2386 +0xfb
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1748 +0x1e65
goroutine 7695 [select, 4257 minutes]:
net/http.(*persistConn).readLoop(0xc0007d2d80)
/usr/local/go/src/net/http/transport.go:2207 +0xd8a
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1747 +0x1e05
goroutine 1450 [select, 4362 minutes]:
net/http.(*persistConn).writeLoop(0xc0007d38c0)
/usr/local/go/src/net/http/transport.go:2386 +0xfb
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1748 +0x1e65
goroutine 171 [chan receive]:
github.com/segmentio/kafka-go.(*conn).run(0xc000080230, 0xc0004c89c0, 0x0)
/go/pkg/mod/github.com/segmentio/kafka-go@v0.4.8/transport.go:1190 +0xb3
created by github.com/segmentio/kafka-go.(*connGroup).connect
/go/pkg/mod/github.com/segmentio/kafka-go@v0.4.8/transport.go:1168 +0xc72
goroutine 166 [select]:
github.com/segmentio/kafka-go.(*connPool).discover(0xc0002db8c0, {0xe2e7d8, 0xc0004c9fc0}, 0xc00003f9e0)
/go/pkg/mod/github.com/segmentio/kafka-go@v0.4.8/transport.go:578 +0x4bd
created by github.com/segmentio/kafka-go.(*Transport).grabPool
/go/pkg/mod/github.com/segmentio/kafka-go@v0.4.8/transport.go:254 +0x5a7
goroutine 1833 [select, 4362 minutes]:
net/http.(*persistConn).readLoop(0xc000d0d680)
/usr/local/go/src/net/http/transport.go:2207 +0xd8a
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1747 +0x1e05
goroutine 1428 [select, 4362 minutes]:
net/http.(*persistConn).readLoop(0xc000a80b40)
/usr/local/go/src/net/http/transport.go:2207 +0xd8a
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1747 +0x1e05
goroutine 47377 [select, 3278 minutes]:
net/http.(*persistConn).writeLoop(0xc000d0c900)
/usr/local/go/src/net/http/transport.go:2386 +0xfb
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1748 +0x1e65
goroutine 1012 [select, 4362 minutes]:
net/http.(*persistConn).readLoop(0xc0001070e0)
/usr/local/go/src/net/http/transport.go:2207 +0xd8a
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1747 +0x1e05
goroutine 1514 [select, 4362 minutes]:
net/http.(*persistConn).readLoop(0xc0002d6fc0)
/usr/local/go/src/net/http/transport.go:2207 +0xd8a
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1747 +0x1e05
goroutine 1005 [select, 4362 minutes]:
net/http.(*persistConn).writeLoop(0xc000106d80)
/usr/local/go/src/net/http/transport.go:2386 +0xfb
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1748 +0x1e65
goroutine 6195 [select, 4285 minutes]:
net/http.(*persistConn).writeLoop(0xc00007c6c0)
/usr/local/go/src/net/http/transport.go:2386 +0xfb
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1748 +0x1e65
goroutine 67884 [select, 2892 minutes]:
net/http.(*persistConn).readLoop(0xc00007c000)
/usr/local/go/src/net/http/transport.go:2207 +0xd8a
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1747 +0x1e05
goroutine 54248 [select, 3139 minutes]:
net/http.(*persistConn).writeLoop(0xc00007cea0)
/usr/local/go/src/net/http/transport.go:2386 +0xfb
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1748 +0x1e65
goroutine 7552 [select, 4259 minutes]:
net/http.(*persistConn).readLoop(0xc0007d2240)
/usr/local/go/src/net/http/transport.go:2207 +0xd8a
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1747 +0x1e05
goroutine 72879 [select, 2793 minutes]:
net/http.(*persistConn).readLoop(0xc000bc6ea0)
/usr/local/go/src/net/http/transport.go:2207 +0xd8a
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1747 +0x1e05
goroutine 1377 [select, 4362 minutes]:
net/http.(*persistConn).writeLoop(0xc0007d2ea0)
/usr/local/go/src/net/http/transport.go:2386 +0xfb
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1748 +0x1e65
goroutine 1392 [select, 4362 minutes]:
net/http.(*persistConn).readLoop(0xc0007d2360)
/usr/local/go/src/net/http/transport.go:2207 +0xd8a
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1747 +0x1e05
goroutine 1756 [select, 4362 minutes]:
net/http.(*persistConn).writeLoop(0xc000d0c480)
/usr/local/go/src/net/http/transport.go:2386 +0xfb
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go

Debug page Info:

Screen Shot 2022-07-04 at 3 20 31 PM 
Redis Info
# Server
redis_version:6.0.16
redis_git_sha1:2858e290
redis_git_dirty:0
redis_build_id:fde4023792433f1f
redis_mode:standalone
os:Linux 4.14.158-129.185.amzn2.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:10.2.1
process_id:11
run_id:cdfc1aa2e5d3c5f152d7e782b2fde8becb9bbd70
tcp_port:0
uptime_in_seconds:4512512
uptime_in_days:52
hz:10
configured_hz:10
lru_clock:12795606
executable:/usr/bin/redis-server
config_file:/tmp/redis.conf
io_threads_active:0

# Clients
connected_clients:3
client_recent_max_input_buffer:2
client_recent_max_output_buffer:0
blocked_clients:2
tracking_clients:0
clients_in_timeout_table:2

# Memory
used_memory:924794
used_memory_human:903.12K
used_memory_rss:3006464
used_memory_rss_human:2.87M
used_memory_peak:2726417
used_memory_peak_human:2.60M
used_memory_peak_perc:33.92%
used_memory_overhead:839470
used_memory_startup:779592
used_memory_dataset:85324
used_memory_dataset_perc:58.76%
allocator_allocated:890564
allocator_active:2968576
allocator_resident:2968576
total_system_memory:8053731328
total_system_memory_human:7.50G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:3.33
allocator_frag_bytes:2078012
allocator_rss_ratio:1.00
allocator_rss_bytes:0
rss_overhead_ratio:1.01
rss_overhead_bytes:37888
mem_fragmentation_ratio:3.38
mem_fragmentation_bytes:2115900
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:51110
mem_aof_buffer:0
mem_allocator:libc
active_defrag_running:0
lazyfree_pending_objects:0

# Persistence
loading:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1656962771
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:409600
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:852
total_commands_processed:51593012
instantaneous_ops_per_sec:0
total_net_input_bytes:7049168493
total_net_output_bytes:434225279
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:30370
evicted_keys:0
keyspace_hits:1301495
keyspace_misses:1850741
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:117
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_reads_processed:51114444
total_writes_processed:51113592
io_threaded_reads_processed:0
io_threaded_writes_processed:0

# Replication
role:master
connected_slaves:0
master_replid:826847018880014ba77227fcc47b9792c8e9a726
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:2623.619967
used_cpu_user:2384.584774
used_cpu_sys_children:16.453156
used_cpu_user_children:88.965642

# Modules

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=168,expires=0,avg_ttl=0

Disk Usage
> df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                  98.3G      2.3G     95.7G   2% /
tmpfs                    64.0M         0     64.0M   0% /dev
tmpfs                     3.8G         0      3.8G   0% /sys/fs/cgroup
/dev/nvme0n1p1           98.3G      2.3G     95.7G   2% /etc/resolv.conf
/dev/nvme0n1p1           98.3G      2.3G     95.7G   2% /etc/hostname
/dev/nvme0n1p1           98.3G      2.3G     95.7G   2% /etc/hosts
shm                      64.0M         0     64.0M   0% /dev/shm
fs-0cf5522094df503bd.efs.us-east-1.amazonaws.com:/
                          8.0E         0      8.0E   0% /var/lib/faktory
tmpfs                     3.8G         0      3.8G   0% /proc/acpi
tmpfs                    64.0M         0     64.0M   0% /proc/kcore
tmpfs                    64.0M         0     64.0M   0% /proc/keys
tmpfs                    64.0M         0     64.0M   0% /proc/latency_stats
tmpfs                    64.0M         0     64.0M   0% /proc/timer_list
tmpfs                    64.0M         0     64.0M   0% /proc/sched_debug
tmpfs                     3.8G         0      3.8G   0% /sys/firmware
mperham commented 2 years ago

Looks like a ton of goroutines are trying to enqueue a job and all stuck waiting for the response. I need to put a read timeout on the IO so that this issue doesn't result in the process deadlocked completely.

github.com/contribsys/faktory/client.readResponse(0xc0002fc000)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:630 +0x45
github.com/contribsys/faktory/client.ok(0xd0)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:593 +0x1d
github.com/contribsys/faktory/client.(*Client).ok(0xc0004c8340, 0xc00073ae60)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:527 +0x25
github.com/contribsys/faktory/client.(*Client).Push(0xc0004c8340, 0xe)
/go/pkg/mod/github.com/contribsys/faktory@v1.6.0/client/client.go:323 +0x74
github.com/merryfield/reward-service/processors.enqueueUpdateProfileJob(0xc0009ac980, {0xc001aab110, 0x24}, {0xe1d760, 0xc0004c8340})
/opt/reward-service/processors/receipt_processing_helpers.go:759 +0x248
github.com/merryfield/reward-service/processors.(*DigitalReceiptProcessor).Process(0xc00023e510, {0xe2e880, 0xc000a298c0}, {0xc0009ac980, 0x1, 0x4})
/opt/reward-service/processors/digital_receipt.go:223 +0x2c6b
mperham commented 2 years ago

^^^^^ You can try compiling your worker against main and see if it works better.

bploetz commented 2 years ago

@mperham I don't know if it helps and/or is relevant, but the few times we've run into this, it always seems to get stuck when jobs are enqueuing other jobs (enqueueUpdateProfileJob() in the thread dumps above). We do that like so:

// in our main(), create a pool and a job client      
mgr := worker.NewManager()

pool, nperr := faktory.NewPool(concurrency + 2)
if nperr != nil {
    log.Panicf("Couldn't create Faktory connection pool: %+v", nperr)
}
mgr.Pool = pool
jobClient, cerr := mgr.Pool.Get()

// We pass the jobClient to the job processing functions, which use it to enqueue another job like so:
jobArgument := make(map[string]interface{})
jobArgument["id"] = receiptID
jobArgs := []interface{}{jobArgument}
retry := 5
updateProfileJob := faktory.NewJob("update-profile", jobArgs...)
updateProfileJob.Queue = "profiles"
updateProfileJob.ReserveFor = 60
updateProfileJob.Retry = &retry

jpperr := jobClient.Push(updateProfileJob)
if jpperr != nil {
    return fmt.Errorf("error enqueueing job: %v", jpperr)
}
mperham commented 2 years ago

The one thing I noticed with that code is your use of the pool. I would urge you to use mgr.Pool.With(function...) instead of Get so that the connection is automatically returned to the pool for you.

https://github.com/contribsys/faktory_worker_go/blob/b8c6acd2d930d8c6bbd5fda6518b47eb68fe7ac0/manager.go#L227

I traced thru the codepath for Push on the server-side and I don't see anything obviously wrong or that would block infinitely unless Redis is not responding for some reason. It's not clear to me which push middleware, if any, are running, that could be another possible source of trouble.

bploetz commented 2 years ago

Duly noted, will try switching over to mgr.Pool.With(function...).

bploetz commented 2 years ago

@mperham this issue is absolutely killing us. Above you suggested using mgr.Pool.With(function...) to have a job enqueue another job. However, the FAQ in faktory_worker_go says:

Inside a job, you can check out a connection from the Pool of Faktory connections using the job helper's With method:

func someFunc(ctx context.Context, args ...interface{}) error {
  help := worker.HelperFor(ctx)
  return help.With(func(cl *faktory.Client) error {
    job := faktory.NewJob("SomeJob", 1, 2, 3)
    return cl.Push(job)
  })
}

Which is the preferred way of having a job enqueue another job? If it's with mgr.Pool.With(function....) like you said above, can you elaborate on how exactly this should be done within a hypothetical someFunc like above? It's not clear to me based on the link you posted to the source.

Thanks.

mperham commented 2 years ago

helper.With is the preferred way since you can get access to it easily within a job func. Maybe the faq needs to be updated.

bploetz commented 1 year ago

@mperham we just released version 1.6.2 which has your fix for this (https://github.com/contribsys/faktory/commit/44668c76d2d2eeca9e7c1b61f8fd9b0c296e53c3) and we're still seeing the same behavior of jobs getting enqueued and not processed.

mperham commented 1 year ago

Use TTIN to get stacks from Faktory and worker processes. Im not sure what else to do.