search

instructlab / instructlab-bot

GitHub bot to assist with the taxonomy contribution workflow
Apache License 2.0
13 stars 16 forks source link

Worker either hung or the API didn't close the connection #286

Closed nerdalert closed 3 months ago

nerdalert commented 3 months ago

We probably need to add a hard timeout on the groutine. Logs:


Apr 22 15:54:19 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[251057]:  not found in the config file. Using default.
Apr 22 15:54:19 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[251057]: INFO 2024-04-22 15:54:19,088 _client.py:1026 HTTP Request: POST https://merlinite-7b-vllm-openai.apps.fmaas-backend.fmaas.res.ibm.com/v1/chat/completions "HTTP/1.1 200 OK"
Apr 22 15:54:21 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: 2024-04-22T15:54:21.266Z        INFO        cmd/generate.go:683        Job took 27s to run
Apr 22 15:54:21 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: 2024-04-22T15:54:21.284Z        INFO        cmd/generate.go:554        Job done        {"job": "182", "pr_number": "730", "work_dir": "/home/fedora/instruct-lab-bot", "origin": "origin", "out_dir": "/home/fedora/instruct-lab-bot/precheck-pr-730-458892bc2b34c55bbe5e4fb3ae827475a54214b7"}
Apr 22 17:13:33 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: 2024-04-22T17:13:33.046Z        INFO        cmd/generate.go:377        Processing job 183        {"job": "183"}
Apr 22 17:13:33 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: 2024-04-22T17:13:33.207Z        INFO        cmd/generate.go:597        Retrying fetching updates, attempt 2/5        {"job": "183", "pr_number": "730", "work_dir": "/home/fedora/instruct-lab-bot", "origin": "origin"}
Apr 22 17:13:35 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: 2024-04-22T17:13:35.354Z        INFO        cmd/generate.go:597        Retrying fetching updates, attempt 3/5        {"job": "183", "pr_number": "730", "work_dir": "/home/fedora/instruct-lab-bot", "origin": "origin"}
Apr 22 17:13:37 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: 2024-04-22T17:13:37.503Z        INFO        cmd/generate.go:597        Retrying fetching updates, attempt 4/5        {"job": "183", "pr_number": "730", "work_dir": "/home/fedora/instruct-lab-bot", "origin": "origin"}
Apr 22 17:13:39 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: 2024-04-22T17:13:39.597Z        INFO        cmd/generate.go:597        Retrying fetching updates, attempt 5/5        {"job": "183", "pr_number": "730", "work_dir": "/home/fedora/instruct-lab-bot", "origin": "origin"}
Apr 22 17:13:43 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: 2024-04-22T17:13:43.973Z        INFO        cmd/generate.go:878        SDG Post Details: &{POST https://datagen-service-predictor-fmaas-rhods-testing.apps.fmaas-devstage-backend.fmaas.res.ibm.com/skill/datagen HTTP/1.1 1 1 map[Accept:[application/json] Content-Type:[application/json]] {{"created_by":"ae2015","mm_model_id":"mistralai/mixtral-8x7b-instruct-v0-1","num_samples":10,"seed_examples":[{"answer":"(A) To apply for public benefit XYZ, you must provide both your full name and vehicle license plate number\n","context":"To apply for public benefit XYZ, provide the following:\n* Your full name and date of birth\n* Your vehicle license plate number\n","question":"Which of the following is true:\n(A) To apply for public benefit XYZ, you must provide both your full name and vehicle license plate number\n(B) To apply for public benefit XYZ, you must provide either your full name or vehicle license plate number\n"},{"answer":"(A) When requesting IRS help, an employee must provide both name and SSN.\n","context":"Form W-2. If you were an employee, you should receive Form W-2 from your employer. You will need the information from this form to prepare your return. See Form W-2 under Credit for Withholding and Estimated Tax for 2023 in chapter 4. Your employer is required to provide or send Form W-2 to you no later than January 31, 2024. If it is mailed, you should allow adequate time to receive it before contacting your employer. If you still don't get the form by early February, the IRS can help you by requesting the form from your employer. When you request IRS help, be prepared to provide the following information.\n* Your name, address (including ZIP code), and phone number.\n* Your SSN.\n* Your dates of employment.\n* Your employer's name, address (including ZIP code), and phone number.\n","question":"Which of the following is true:\n(A) When requesting IRS help, an employee must provide both name and SSN.\n(B) When requesting IRS help, an employee must provide either name or SSN.\n"},{"answer":"(A) If your address is outside the U.S., you must enter both foreign country name and foreign postal code.\n","context":"Foreign address. If your address is outside the United States or its territories, enter the city name on the appropriate line of your Form 1040 or 1040-SR. Don't enter any other information on that line, but also complete the spaces below that line.\n1. Foreign country name.\n2. Foreign province/state/county.\n3. Foreign postal code.\nDon't abbreviate the country name. Follow the country's practice for entering the postal code and the name of the province, county, or state.\n","question":"Which of the following is true:\n(A) If your address is outside the U.S., you must enter both foreign country name and foreign postal code.\n(B) If your address is outside the U.S., you must enter either foreign country name or foreign postal code.\n"},{"answer":"(A) To transfer the vehicle's title, you must have both your driver's license number and vehicle license plate number.\n","context":"TITLE TRANSFERS\nWhen you buy a vehicle, you need to transfer the vehicle's title to establish yourself as the new legal owner.\nBefore you begin:\n1. Have the California Certificate of Title with you. Make sure the title has been signed by the buyer(s), seller(s), and lienholder (if applicable).\nNote: If you do not have the title, complete an Application for Replacement or Transfer of Title (REG 227).\n2. Have the following:\n* Your driver's license number\n* Vehicle license plate number\n* Vehicle identification number (VIN)\n* Legal owner (or lienholder) name and address\n* Vehicle make, model, and year\n* Purchase date and price\n3. Be able to pay the transfer fee. All transfer fees are the responsibility of the buyer.\n4. Allow 30 days from the day DMV receives your documents to process your title transfer.\n","question":"Which of the following is true:\n(A) To transfer the vehicle's title, you must have both your driver's license number and vehicle license plate number.\n(B) To transfer the vehicle's title, you must have either your driver's license number or vehicle license plate number.\n"},{"answer":"(A) When you arrive at a DMV office, you will need both correct fee(s) for your application and residency document.\n","context":"To apply for an ID, you must complete a Driver License and Identification Card application and pay any fees. For current ID fees, visit www.dmv.ca.gov.\n1.  Before visiting a DMV office:\n* Gather the identity, residency, and SSN documents you need to apply for an ID card.\n* Effective April 2018 DMV will begin offering an online DL and ID application process. Applicants will complete their electronic application before visiting DMV. Be sure to bring your application confirmation with you to your office visit.\n* Make an appointment before visiting a DMV office. Appointments can be made online at www.dmv.ca.gov or by calling 1-800-777-0133. You may also verify payment options available at the DMV office.\n2.  When you arrive at a DMV office, you will need:\n* Correct fee(s) for your application.\n* Original or certified documents(s) establishing identity, birth date, true full name.\n* Original document establishing proof of SSN if applying for a REAL ID Compliant ID.\n* Residency Document.\nThe DMV representative may give you a number and/or direct you to wait. When it is your turn, you will be called to a window to process your application.\n","question":"Which of the following is true:\n(A) When you arrive at a DMV office, you will need both correct fee(s) for your application and residency document.\n(B) When you arrive at a DMV office, you will need either correct fee(s) for your application or residency document.\n"}],"task_description":"Understand when a bullet list of items means that the items are needed jointly rather than separately.\n"}} 0x6c14c0 5587 [] false datagen-service-predictor-fmaas-rhods-testing.apps.fmaas-devstage-backend.fmaas.res.ibm.com map[] map[] <nil> map[]   <nil> <nil> <nil> {{}}}
Apr 22 18:45:05 ip-172-31-13-79.us-east-2.compute.internal systemd[1]: Stopping labbotworker.service - Instruct Lab GitHub Bot Worker...
Apr 22 18:45:05 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: 2024-04-22T18:45:05.300Z        INFO        cmd/generate.go:187        Shutting down
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal systemd[1]: labbotworker.service: State 'stop-sigterm' timed out. Aborting.
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal systemd[1]: labbotworker.service: Killing process 236324 (instruct-lab-bo) with signal SIGABRT.
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: SIGABRT: abort
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: PC=0x46f7c1 m=0 sigcode=0
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: goroutine 0 [idle]:
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.futex()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7ffd274be410 sp=0x7ffd274be408 pc=0x46f7c1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.futexsleep(0x7ffd274be488?, 0x442456?, 0x7ffd274be488?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7ffd274be460 sp=0x7ffd274be410 pc=0x437490
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.notesleep(0x1678d48)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/lock_futex.go:160 +0x87 fp=0x7ffd274be498 sp=0x7ffd274be460 pc=0x410a27
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.mPark(...)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/proc.go:1634
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.stoplockedm()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/proc.go:2775 +0x73 fp=0x7ffd274be4f0 sp=0x7ffd274be498 pc=0x442633
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.schedule()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/proc.go:3556 +0x3a fp=0x7ffd274be528 sp=0x7ffd274be4f0 pc=0x44447a
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.park_m(0xc0001e1380?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/proc.go:3740 +0x11f fp=0x7ffd274be570 sp=0x7ffd274be528 pc=0x4449ff
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.mcall()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x4e fp=0x7ffd274be588 sp=0x7ffd274be570 pc=0x46b9ae
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: goroutine 1 [semacquire, 4538 minutes]:
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.gopark(0xc0001873c0?, 0x0?, 0x0?, 0xe0?, 0x41f216?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc0005053a0 sp=0xc000505380 pc=0x43e08e
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.goparkunlock(...)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/proc.go:404
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.semacquire1(0xc00028e008, 0x18?, 0x1, 0x0, 0x2b?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/sema.go:160 +0x218 fp=0xc000505408 sp=0xc0005053a0 pc=0x44ebf8
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: sync.runtime_Semacquire(0xc00029c018?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/sema.go:62 +0x25 fp=0xc000505440 sp=0xc000505408 pc=0x469c05
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: sync.(*WaitGroup).Wait(0xc00007e5a0?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/sync/waitgroup.go:116 +0x48 fp=0xc000505468 sp=0xc000505440 pc=0x4797a8
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: github.com/instructlab/instructlab-bot/worker/cmd.glob..func1(0x1671be0, {0xf4989a?, 0x4?, 0xf49756?})
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /home/fedora/worker-intruct-lab-bot/instruct-lab-bot/worker/cmd/generate.go:200 +0x765 fp=0xc000505920 sp=0xc000505468 pc=0xd6ec25
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: github.com/spf13/cobra.(*Command).execute(0x1671be0, {0xc0002144d0, 0x7, 0x7})
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /home/fedora/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:987 +0xaa3 fp=0xc000505ac0 sp=0xc000505920 pc=0xc36543
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: github.com/spf13/cobra.(*Command).ExecuteC(0x1671ec0)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /home/fedora/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115 +0x3ff fp=0xc000505b98 sp=0xc000505ac0 pc=0xc36e1f
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: github.com/spf13/cobra.(*Command).Execute(...)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /home/fedora/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: github.com/instructlab/instructlab-bot/worker/cmd.Execute()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /home/fedora/worker-intruct-lab-bot/instruct-lab-bot/worker/cmd/root.go:35 +0x1a fp=0xc000505be0 sp=0xc000505b98 pc=0xd7975a
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: main.main()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /home/fedora/worker-intruct-lab-bot/instruct-lab-bot/worker/main.go:26 +0x22a fp=0xc000505f40 sp=0xc000505be0 pc=0xd7b7ea
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.main()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/proc.go:267 +0x2bb fp=0xc000505fe0 sp=0xc000505f40 pc=0x43dc3b
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.goexit()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000505fe8 sp=0xc000505fe0 pc=0x46d9a1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: goroutine 2 [force gc (idle), 3 minutes]:
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.gopark(0xa0d11d07af8b6?, 0x0?, 0x0?, 0x0?, 0x0?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc000062fa8 sp=0xc000062f88 pc=0x43e08e
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.goparkunlock(...)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/proc.go:404
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.forcegchelper()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/proc.go:322 +0xb3 fp=0xc000062fe0 sp=0xc000062fa8 pc=0x43df13
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.goexit()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000062fe8 sp=0xc000062fe0 pc=0x46d9a1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: created by runtime.init.6 in goroutine 1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/proc.go:310 +0x1a
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: goroutine 3 [GC sweep wait]:
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.gopark(0x1677901?, 0x0?, 0x0?, 0x0?, 0x0?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc000063778 sp=0xc000063758 pc=0x43e08e
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.goparkunlock(...)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/proc.go:404
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.bgsweep(0x0?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/mgcsweep.go:321 +0xdf fp=0xc0000637c8 sp=0xc000063778 pc=0x429d1f
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.gcenable.func1()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/mgc.go:200 +0x25 fp=0xc0000637e0 sp=0xc0000637c8 pc=0x41ee85
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.goexit()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000637e8 sp=0xc0000637e0 pc=0x46d9a1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: created by runtime.gcenable in goroutine 1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/mgc.go:200 +0x66
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: goroutine 4 [GC scavenge wait]:
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.gopark(0x1678200?, 0xe5973?, 0x0?, 0x0?, 0x0?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc000063f70 sp=0xc000063f50 pc=0x43e08e
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.goparkunlock(...)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/proc.go:404
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.(*scavengerState).park(0x1678200)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000063fa0 sp=0xc000063f70 pc=0x4275a9
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.bgscavenge(0x0?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc000063fc8 sp=0xc000063fa0 pc=0x427b59
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.gcenable.func2()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/mgc.go:201 +0x25 fp=0xc000063fe0 sp=0xc000063fc8 pc=0x41ee25
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.goexit()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000063fe8 sp=0xc000063fe0 pc=0x46d9a1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: created by runtime.gcenable in goroutine 1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/mgc.go:201 +0xa5
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: goroutine 5 [finalizer wait, 90 minutes]:
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.gopark(0x0?, 0xc000024108?, 0x0?, 0x0?, 0x1000000010?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc000078e20 sp=0xc000078e00 pc=0x43e08e
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.runfinq()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/mfinal.go:193 +0x107 fp=0xc000078fe0 sp=0xc000078e20 pc=0x41dea7
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.goexit()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000078fe8 sp=0xc000078fe0 pc=0x46d9a1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: created by runtime.createfing in goroutine 1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/mfinal.go:163 +0x3d
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: goroutine 6 [select, 4538 minutes, locked to thread]:
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.gopark(0xc0000647a8?, 0x2?, 0x29?, 0xe3?, 0xc0000647a4?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc000064638 sp=0xc000064618 pc=0x43e08e
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.selectgo(0xc0000647a8, 0xc0000647a0, 0x0?, 0x0, 0x0?, 0x1)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/select.go:327 +0x725 fp=0xc000064758 sp=0xc000064638 pc=0x44dbc5
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.ensureSigM.func1()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/signal_unix.go:1014 +0x19f fp=0xc0000647e0 sp=0xc000064758 pc=0x464cbf
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.goexit()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000647e8 sp=0xc0000647e0 pc=0x46d9a1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: created by runtime.ensureSigM in goroutine 1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/signal_unix.go:997 +0xc8
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: goroutine 18 [syscall]:
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.notetsleepg(0xffffffffffffffff?, 0xc00005e728?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/lock_futex.go:236 +0x29 fp=0xc00005e7a0 sp=0xc00005e768 pc=0x410d09
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: os/signal.signal_recv()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/sigqueue.go:152 +0x29 fp=0xc00005e7c0 sp=0xc00005e7a0 pc=0x46a369
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: os/signal.loop()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/os/signal/signal_unix.go:23 +0x13 fp=0xc00005e7e0 sp=0xc00005e7c0 pc=0x6ea833
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.goexit()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00005e7e8 sp=0xc00005e7e0 pc=0x46d9a1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: created by os/signal.Notify.func1.1 in goroutine 1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/os/signal/signal.go:151 +0x1f
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: goroutine 34 [chan send, 70 minutes]:
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc0002bde08 sp=0xc0002bdde8 pc=0x43e08e
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.chansend(0xc00004b3e0, 0xc0002bdf48, 0x1, 0xc000098280?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/chan.go:259 +0x3a5 fp=0xc0002bde78 sp=0xc0002bde08 pc=0x40ac85
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.chansend1(0xc0004286f0?, 0xc000428708?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/chan.go:145 +0x17 fp=0xc0002bdea8 sp=0xc0002bde78 pc=0x40a8d7
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: github.com/instructlab/instructlab-bot/worker/cmd.glob..func1.2(0x0?, 0xc00004b440)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /home/fedora/worker-intruct-lab-bot/instruct-lab-bot/worker/cmd/generate.go:178 +0x25f fp=0xc0002bdfc0 sp=0xc0002bdea8 pc=0xd6f3df
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: github.com/instructlab/instructlab-bot/worker/cmd.glob..func1.7()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /home/fedora/worker-intruct-lab-bot/instruct-lab-bot/worker/cmd/generate.go:181 +0x2b fp=0xc0002bdfe0 sp=0xc0002bdfc0 pc=0xd6f14b
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.goexit()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0002bdfe8 sp=0xc0002bdfe0 pc=0x46d9a1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: created by github.com/instructlab/instructlab-bot/worker/cmd.glob..func1 in goroutine 1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /home/fedora/worker-intruct-lab-bot/instruct-lab-bot/worker/cmd/generate.go:159 +0x545
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: goroutine 36 [select, 92 minutes]:
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.gopark(0xc0003a7078?, 0x6?, 0xdc?, 0x89?, 0xc0003a6e8c?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc0003a6d10 sp=0xc0003a6cf0 pc=0x43e08e
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.selectgo(0xc0003a7078, 0xc0003a6e80, 0xf4fef2?, 0x0, 0xc0003a6f38?, 0x1)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/select.go:327 +0x725 fp=0xc0003a6e30 sp=0xc0003a6d10 pc=0x44dbc5
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: net/http.(*persistConn).roundTrip(0xc000143200, 0xc00029e5c0)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/net/http/transport.go:2652 +0x979 fp=0xc0003a70e8 sp=0xc0003a6e30 pc=0x6d9979
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: net/http.(*Transport).roundTrip(0xc0002cb040, 0xc0001bac00)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/net/http/transport.go:604 +0x79a fp=0xc0003a7310 sp=0xc0003a70e8 pc=0x6cdcfa
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: net/http.(*Transport).RoundTrip(0x452d2f?, 0x10b2bc0?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/net/http/roundtrip.go:17 +0x13 fp=0xc0003a7330 sp=0xc0003a7310 pc=0x6c4973
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: net/http.send(0xc0001bac00, {0x10b2bc0, 0xc0002cb040}, {0xc000016001?, 0x1751?, 0x0?})
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/net/http/client.go:260 +0x606 fp=0xc0003a7560 sp=0xc0003a7330 pc=0x69a6a6
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: net/http.(*Client).send(0xc00044c600, 0xc0001bac00, {0x4783aa?, 0xc0000a2bd0?, 0x0?})
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/net/http/client.go:181 +0x98 fp=0xc0003a75d8 sp=0xc0003a7560 pc=0x699f38
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: net/http.(*Client).do(0xc00044c600, 0xc0001bac00)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/net/http/client.go:724 +0x912 fp=0xc0003a77e8 sp=0xc0003a75d8 pc=0x69c212
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: net/http.(*Client).Do(...)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/net/http/client.go:590
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: github.com/instructlab/instructlab-bot/worker/cmd.(*Worker).datagenSvc(0xc0000a6640, {0xc00004c010, 0x1, 0x1?}, {0xc0004ab6e0, 0x55}, 0x11?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /home/fedora/worker-intruct-lab-bot/instruct-lab-bot/worker/cmd/generate.go:880 +0x5ff fp=0xc0003a79f8 sp=0xc0003a77e8 pc=0xd7643f
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: github.com/instructlab/instructlab-bot/worker/cmd.(*Worker).processJob(0xc0000a6640)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /home/fedora/worker-intruct-lab-bot/instruct-lab-bot/worker/cmd/generate.go:530 +0x15e5 fp=0xc0003a7ed0 sp=0xc0003a79f8 pc=0xd72985
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: github.com/instructlab/instructlab-bot/worker/cmd.glob..func1.4(0x0?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /home/fedora/worker-intruct-lab-bot/instruct-lab-bot/worker/cmd/generate.go:196 +0x172 fp=0xc0003a7fc8 sp=0xc0003a7ed0 pc=0xd6ee72
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: github.com/instructlab/instructlab-bot/worker/cmd.glob..func1.9()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /home/fedora/worker-intruct-lab-bot/instruct-lab-bot/worker/cmd/generate.go:198 +0x27 fp=0xc0003a7fe0 sp=0xc0003a7fc8 pc=0xd6ecc7
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.goexit()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0003a7fe8 sp=0xc0003a7fe0 pc=0x46d9a1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: created by github.com/instructlab/instructlab-bot/worker/cmd.glob..func1 in goroutine 1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /home/fedora/worker-intruct-lab-bot/instruct-lab-bot/worker/cmd/generate.go:192 +0x755
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: goroutine 11 [GC worker (idle), 3 minutes]:
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.gopark(0xa0d11d087a7ba?, 0x3?, 0xfc?, 0xca?, 0xc0002967a8?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc000296750 sp=0xc000296730 pc=0x43e08e
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.gcBgMarkWorker()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/mgc.go:1295 +0xe5 fp=0xc0002967e0 sp=0xc000296750 pc=0x420a05
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.goexit()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0002967e8 sp=0xc0002967e0 pc=0x46d9a1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: created by runtime.gcBgMarkStartWorkers in goroutine 20
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/mgc.go:1219 +0x1c
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: goroutine 21 [GC worker (idle), 92 minutes]:
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.gopark(0xa081f6e705537?, 0x3?, 0xf0?, 0xe5?, 0xc0001d6f00?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc000297750 sp=0xc000297730 pc=0x43e08e
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.gcBgMarkWorker()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/mgc.go:1295 +0xe5 fp=0xc0002977e0 sp=0xc000297750 pc=0x420a05
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.goexit()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0002977e8 sp=0xc0002977e0 pc=0x46d9a1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: created by runtime.gcBgMarkStartWorkers in goroutine 20
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/mgc.go:1219 +0x1c
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: goroutine 12 [GC worker (idle), 92 minutes]:
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.gopark(0xa081eef50e121?, 0x3?, 0xcb?, 0x9a?, 0x0?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc000064f50 sp=0xc000064f30 pc=0x43e08e
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.gcBgMarkWorker()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/mgc.go:1295 +0xe5 fp=0xc000064fe0 sp=0xc000064f50 pc=0x420a05
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.goexit()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000064fe8 sp=0xc000064fe0 pc=0x46d9a1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: created by runtime.gcBgMarkStartWorkers in goroutine 20
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/mgc.go:1219 +0x1c
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: goroutine 13 [GC worker (idle), 92 minutes]:
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.gopark(0xa081f6e7057a9?, 0x3?, 0xa3?, 0xa9?, 0x0?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc000065750 sp=0xc000065730 pc=0x43e08e
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.gcBgMarkWorker()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/mgc.go:1295 +0xe5 fp=0xc0000657e0 sp=0xc000065750 pc=0x420a05
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.goexit()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000657e8 sp=0xc0000657e0 pc=0x46d9a1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: created by runtime.gcBgMarkStartWorkers in goroutine 20
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/mgc.go:1219 +0x1c
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: goroutine 476 [IO wait, 92 minutes]:
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.gopark(0x9abbad317f79a4cf?, 0xb?, 0x0?, 0x0?, 0x9?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc0002be5c8 sp=0xc0002be5a8 pc=0x43e08e
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.netpollblock(0x48e218?, 0x408b66?, 0x0?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/netpoll.go:564 +0xf7 fp=0xc0002be600 sp=0xc0002be5c8 pc=0x436857
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: internal/poll.runtime_pollWait(0x7f6d9a2c4960, 0x72)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/netpoll.go:343 +0x85 fp=0xc0002be620 sp=0xc0002be600 pc=0x468385
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: internal/poll.(*pollDesc).wait(0xc000022180?, 0xc00027c000?, 0x0)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0002be648 sp=0xc0002be620 pc=0x512f07
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: internal/poll.(*pollDesc).waitRead(...)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: internal/poll.(*FD).Read(0xc000022180, {0xc00027c000, 0x1000, 0x1000})
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/internal/poll/fd_unix.go:164 +0x27a fp=0xc0002be6e0 sp=0xc0002be648 pc=0x5141fa
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: net.(*netFD).Read(0xc000022180, {0xc00027c000?, 0xc00027c724?, 0x71a?})
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/net/fd_posix.go:55 +0x25 fp=0xc0002be728 sp=0xc0002be6e0 pc=0x5c1225
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: net.(*conn).Read(0xc0000664c0, {0xc00027c000?, 0x0?, 0xc0004048b8?})
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/net/net.go:179 +0x45 fp=0xc0002be770 sp=0xc0002be728 pc=0x5d1425
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: net.(*TCPConn).Read(0xc0002be808?, {0xc00027c000?, 0xc00039c030?, 0x18?})
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         <autogenerated>:1 +0x25 fp=0xc0002be7a0 sp=0xc0002be770 pc=0x5e2405
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: crypto/tls.(*atLeastReader).Read(0xc00039c030, {0xc00027c000?, 0xc00039c030?, 0x0?})
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/crypto/tls/conn.go:805 +0x3b fp=0xc0002be7e8 sp=0xc0002be7a0 pc=0x62041b
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: bytes.(*Buffer).ReadFrom(0xc0004049a8, {0x10b3920, 0xc00039c030})
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/bytes/buffer.go:211 +0x98 fp=0xc0002be840 sp=0xc0002be7e8 pc=0x481cb8
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: crypto/tls.(*Conn).readFromUntil(0xc000404700, {0x10b56e0?, 0xc0000664c0}, 0x8e1?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/crypto/tls/conn.go:827 +0xde fp=0xc0002be880 sp=0xc0002be840 pc=0x6205fe
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: crypto/tls.(*Conn).readRecordOrCCS(0xc000404700, 0x0)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/crypto/tls/conn.go:625 +0x250 fp=0xc0002bec20 sp=0xc0002be880 pc=0x61dbd0
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: crypto/tls.(*Conn).readRecord(...)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/crypto/tls/conn.go:587
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: crypto/tls.(*Conn).Read(0xc000404700, {0xc000035000, 0x1000, 0xe8c811?})
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/crypto/tls/conn.go:1369 +0x158 fp=0xc0002bec90 sp=0xc0002bec20 pc=0x623e98
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: net/http.(*persistConn).Read(0xc000143200, {0xc000035000?, 0x40a805?, 0x60?})
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/net/http/transport.go:1954 +0x4a fp=0xc0002becf0 sp=0xc0002bec90 pc=0x6d618a
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: bufio.(*Reader).fill(0xc0003af1a0)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/bufio/bufio.go:113 +0x103 fp=0xc0002bed28 sp=0xc0002becf0 pc=0x66b563
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: bufio.(*Reader).Peek(0xc0003af1a0, 0x1)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/bufio/bufio.go:151 +0x53 fp=0xc0002bed48 sp=0xc0002bed28 pc=0x66b693
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: net/http.(*persistConn).readLoop(0xc000143200)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/net/http/transport.go:2118 +0x1b9 fp=0xc0002befc8 sp=0xc0002bed48 pc=0x6d6f79
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: net/http.(*Transport).dialConn.func5()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/net/http/transport.go:1776 +0x25 fp=0xc0002befe0 sp=0xc0002befc8 pc=0x6d57a5
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.goexit()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0002befe8 sp=0xc0002befe0 pc=0x46d9a1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: created by net/http.(*Transport).dialConn in goroutine 472
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/net/http/transport.go:1776 +0x169f
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: goroutine 477 [select, 92 minutes]:
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.gopark(0xc0002b9f90?, 0x2?, 0xe0?, 0x9d?, 0xc0002b9f1c?)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc0002b9db8 sp=0xc0002b9d98 pc=0x43e08e
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.selectgo(0xc0002b9f90, 0xc0002b9f18, 0xc00029e780?, 0x0, 0xc00044d890?, 0x1)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/select.go:327 +0x725 fp=0xc0002b9ed8 sp=0xc0002b9db8 pc=0x44dbc5
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: net/http.(*persistConn).writeLoop(0xc000143200)
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/net/http/transport.go:2421 +0xe5 fp=0xc0002b9fc8 sp=0xc0002b9ed8 pc=0x6d8945
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: net/http.(*Transport).dialConn.func6()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/net/http/transport.go:1777 +0x25 fp=0xc0002b9fe0 sp=0xc0002b9fc8 pc=0x6d5745
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: runtime.goexit()
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0002b9fe8 sp=0xc0002b9fe0 pc=0x46d9a1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: created by net/http.(*Transport).dialConn in goroutine 472
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]:         /usr/lib/golang/src/net/http/transport.go:1777 +0x16f1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: rax    0xca
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: rbx    0x0
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: rcx    0x46f7c3
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: rdx    0x0
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: rdi    0x1678d48
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: rsi    0x80
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: rbp    0x7ffd274be450
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: rsp    0x7ffd274be408
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: r8     0x0
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: r9     0x0
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: r10    0x0
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: r11    0x286
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: r12    0x4448e0
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: r13    0xc0000647a4
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: r14    0x16785e0
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: r15    0x1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: rip    0x46f7c1
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: rflags 0x286
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: cs     0x33
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: fs     0x0
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[236324]: gs     0x0
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal systemd[1]: labbotworker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal systemd[1]: labbotworker.service: Failed with result 'timeout'.
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal systemd[1]: Stopped labbotworker.service - Instruct Lab GitHub Bot Worker.
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal systemd[1]: labbotworker.service: Consumed 2min 17.260s CPU time.
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal systemd[1]: Started labbotworker.service - Instruct Lab GitHub Bot Worker.
Apr 22 18:45:50 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[251701]: 2024-04-22T18:45:50.471Z        INFO        cmd/generate.go:133        Starting generate worker
Apr 22 18:45:51 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[251701]: 2024-04-22T18:45:51.474Z        INFO        cmd/generate.go:377        Processing job 185        {"job": "185"}
Apr 22 18:45:51 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[251701]: 2024-04-22T18:45:51.623Z        INFO        cmd/generate.go:597        Retrying fetching updates, attempt 2/5        {"job": "185", "pr_number": "737", "work_dir": "/home/fedora/instruct-lab-bot", "origin": "origin"}
Apr 22 18:45:53 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[251701]: 2024-04-22T18:45:53.697Z        INFO        cmd/generate.go:597        Retrying fetching updates, attempt 3/5        {"job": "185", "pr_number": "737", "work_dir": "/home/fedora/instruct-lab-bot", "origin": "origin"}
Apr 22 18:45:55 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[251701]: 2024-04-22T18:45:55.766Z        INFO        cmd/generate.go:597        Retrying fetching updates, attempt 4/5        {"job": "185", "pr_number": "737", "work_dir": "/home/fedora/instruct-lab-bot", "origin": "origin"}
Apr 22 18:45:57 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[251701]: 2024-04-22T18:45:57.837Z        INFO        cmd/generate.go:597        Retrying fetching updates, attempt 5/5        {"job": "185", "pr_number": "737", "work_dir": "/home/fedora/instruct-lab-bot", "origin": "origin"}
Apr 22 18:46:02 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[251701]: 2024-04-22T18:46:02.461Z        INFO        cmd/generate.go:330        Running the precheck command: /usr/local/bin/ilab chat --quick-question can you solve the following riddle? Two turkeys in front of the turkey, two turkeys behind the turkey, tell me how many total turkeys? --tls-insecure --endpoint-url https://merlinite-7b-vllm-openai.apps.fmaas-backend.fmaas.res.ibm.com/v1/ --model /shared_model_storage/transformers_cache/models--ibm--merlinite-7b/snapshots/233d12759d5bb9344231dafdb51310ec19d79c0e
nerdalert commented 3 months ago

Turns out the job was just taking a long time (2 hours to run)

nerdalert commented 3 months ago

This was a result of an SDG job with context taking 2 hours. Worker was fine. Raised context issue with backend.