Schedules Stuck on "Processing" for All Servers

[dparkinson101]

Current Behavior

Server schedules are not performing any commands or power actions actions and are getting stuck on "Processing" stage. Restarting pteroq or doing php artisan queue:restart does not effect this behaviour.

Expected Behavior

All schedules should complete tasks in order and be able to send commands / powr actions to the server.

Steps to Reproduce

1. Setup Minecraft (Paper) Egg on Server
2. Create CRON Job Schedule through the "Schedule" tab on the panel called "Restart"
3. Add Task to do command "say Server Restarting in 5 Mins"
4. Add Task (with delay of 300) to do a power action for restarting the server
5. Set Schedule to run 1 minute from current time. (At 4:00AM set Schedule to run at 4:01 AM) (Note: This is also seen with manual runs of the schedule so setting the time doesn't seem to be necessary)
6. Observe Schedule getting stuck in the "processing state" when it runs
7. Delete Schedule
8. Do systemctl restart pteroq in pterodactyl server CLI
9. Repeat previous steps 1-7 and see same behaviour
10. Do cd /var/www/pterodactyl/ && php artisan queue:restart
11. Repeat previous steps 1-7 and see same behaviour

Panel Version

1.7.0

Wings Version

1.6.1

Games and/or Eggs Affected

Minecraft (Paper)

Docker Image

ghcr.io/pterodactyl/yolks:java_17

Error Logs

https://ptero.co/hywoxokale
(Nothing in wings seems to be going wrong at the times when I test the scheduler)

No laravel error logs were produced from the above process at current time and date.

Did systemctl status pteroq and no errors or warnings were observed.

Is there an existing issue for this?

• [X] I have searched the existing issues before opening this issue.
• [X] I have provided all relevant details, including the specific game and Docker images I am using if this issue is related to running a server.
• [X] I have checked in the Discord server and believe this is a bug with the software, and not a configuration issue with my specific system.

[dparkinson101]

Further investigation shows that when there is only one command task in the schedule it will do that one command but remain stuck in the processing phase and not trigger ever again. This is only true for command tasks and not power actions which never trigger. Also I am hosting Pterodactyl on CentOS 8 which I forgot to mention in the original ticket.

[dparkinson101]

Further testing proves this issue is on my other servers as well, for instance I have found schedules stuck on "processing" on my Vintage Story server. Tried updating centos 8 to centos 8 rolling to fix this but had no effect. Edited title to reflect how this issue actually manifests.

[parkervcp]

generally anything getting stuck in processing is due to the pteroq service locking up. You should check that

edit: after more than the cursory glance I gave this I see you say you have restarted the pteroq.

what php version are you on?

[dparkinson101]

I'm running PHP version 8.0.17 and Composer version 2.1.3

[DaneEveritt]

Can you please provide a little more detail to your steps to reproduce? When you say "Observe Schedule getting stuck in the "processing state"" what does that mean? What does "Set Schedule to do time 1 minute from current time." mean — are you setting the cron to run every minute, or just setting it to a specific run-time that is in the next minute?

Does this happen on manual runs or only automated ones (e.g. if you manually run the schedule ignoring the cron, does it work?).

Does this happen for any schedule combination, or is there a specific set of things that needs to be true?

[dparkinson101]

Sorry I was a unclear with those step descriptions. Wrote them at 2AM so they're riddled with spelling / grammar mistakes. I updated the original reproduction steps to reflect my current understanding of the issue. I've tried setting the schedule 1-5 mins ahead of the current time to run and also tried a manual run. Both seem to trigger this behaviour for me. It also seems to trigger for any schedule I make on any server I make. Doesn't seem to be linked to any particular combination but I did notice this started happening a few days after updating the panel & wings about a month ago.

[dparkinson101]

Rescently looked at my systemctl status today when trying again and it looks like the RunTaskJob is failing

● pteroq.service - Pterodactyl Queue Worker
   Loaded: loaded (/etc/systemd/system/pteroq.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2022-03-30 12:51:19 EDT; 16min ago
 Main PID: 80406 (php)
    Tasks: 3 (limit: 363210)
   Memory: 40.2M
   CGroup: /system.slice/pteroq.service
           ├─80406 /usr/bin/php /var/www/pterodactyl/artisan queue:work --queue=high,standard,low --sleep=3 --tries=3
           ├─80414 /bin/bash /usr/sbin/sendmail -bs
           └─80420 cat

Mar 30 12:51:19 localhost.localdomain systemd[1]: Started Pterodactyl Queue Worker.
Mar 30 12:51:20 localhost.localdomain php[80406]: [2022-03-30 17:51:20][yQEK6SFbN8Rz9iPyAkEkkxOzWKH9Mql4] Processing: Pterodactyl\Jobs\Schedule\RunTaskJob
Mar 30 12:51:20 localhost.localdomain php[80406]: [2022-03-30 17:51:20][yQEK6SFbN8Rz9iPyAkEkkxOzWKH9Mql4] Failed:     Pterodactyl\Jobs\Schedule\RunTaskJob
Mar 30 12:51:20 localhost.localdomain php[80406]: [2022-03-30 17:51:20][sAh93AoDmWZJUfqNP9HoCEQ3eY8rtzuj] Processing: Pterodactyl\Notifications\ServerInstalled

[BakiDance]

I'm also affected by this issue. I have two schedules, both stuck processing on Ubuntu 20.04 [1.7.0 panel, 1.6.1 wings] with the same image/eggs as above.

One of them restarts the server after running some server commands, and the other makes a backup of the server. I was able to run the backup schedule last night, manually, and it worked. However, I checked this morning and neither schedule has run, and both are stuck on "processing".

I checked the status of pteroq.service and noticed it was inactive (dead). After restarting the service, it seems to have caught up.

● pteroq.service - Pterodactyl Queue Worker
     Loaded: loaded (/etc/systemd/system/pteroq.service; disabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-03-31 15:31:00 UTC; 24s ago
   Main PID: 291572 (php)
      Tasks: 1 (limit: 76979)
     Memory: 39.5M
        CPU: 314ms
     CGroup: /system.slice/pteroq.service
             └─291572 /usr/bin/php /var/www/pterodactyl/artisan queue:work --queue=high,standard,low --sleep=3 --tries=3

Mar 31 15:31:17 localhost php[291572]: [2022-03-31 11:31:17][23] Processing: Pterodactyl\Jobs\Schedule\RunTaskJob
Mar 31 15:31:17 localhost php[291572]: [2022-03-31 11:31:17][23] Failed:     Pterodactyl\Jobs\Schedule\RunTaskJob
Mar 31 15:31:17 localhost php[291572]: [2022-03-31 11:31:17][24] Processing: Pterodactyl\Jobs\Schedule\RunTaskJob
Mar 31 15:31:18 localhost php[291572]: [2022-03-31 11:31:18][24] Processed:  Pterodactyl\Jobs\Schedule\RunTaskJob
Mar 31 15:31:18 localhost php[291572]: [2022-03-31 11:31:18][25] Processing: Pterodactyl\Jobs\Schedule\RunTaskJob
Mar 31 15:31:18 localhost php[291572]: [2022-03-31 11:31:18][25] Processed:  Pterodactyl\Jobs\Schedule\RunTaskJob
Mar 31 15:31:21 localhost php[291572]: [2022-03-31 11:31:21][26] Processing: Pterodactyl\Jobs\Schedule\RunTaskJob
Mar 31 15:31:21 localhost php[291572]: [2022-03-31 11:31:21][26] Processed:  Pterodactyl\Jobs\Schedule\RunTaskJob
Mar 31 15:31:24 localhost php[291572]: [2022-03-31 11:31:24][28] Processing: Pterodactyl\Jobs\Schedule\RunTaskJob
Mar 31 15:31:24 localhost php[291572]: [2022-03-31 11:31:24][28] Processed:  Pterodactyl\Jobs\Schedule\RunTaskJob

[dparkinson101]

Looks like I narrowed down my issue a bit more, whenever I restart pteroq.service I get this crop up and it stays processing no matter what.

Apr 03 18:10:02 localhost.localdomain systemd[1]: Started Pterodactyl Queue Worker.
Apr 03 18:10:02 localhost.localdomain php[416542]: [2022-04-03 23:10:02][ZFAiZdMfuyviaTSXBk85obomLcBFYYaG] Processing: Pterodactyl\Notifications\ServerInstalled

Is there a way I can see which server this process is running for and disable the server to see if that resolves my issue?

Also @BakiDance a work around for that (since from my understanding it is a task failing once stopping the schedule from completing and then not rerunning) is to enable "Continue on Failure" for all tasks in that schedule which might stop them getting stuck in processing. (My issue is different from yours as restarting pteroq does not help unfortunately 😢 )

[Pomdre]

I had the same problem. Looks like it has to do with using coomand send and not power acction to restart the server. A little weird indeed

[ghost]

I am also having this issue on panel v1.8.1.

[ndisalvio3]

Can confirm happens on panel v1.8.1

[DaneEveritt]

@Pomdre @MrSir0814 @ndisalvio3 -- please provide your logs from pteroq itself so we can see if it is running properly and what tasks it appears to hang on.

[masterhacka]

Same on panel v1.9.0 and v1.9.1

[mihaigsm2003]

same to me, stuck in "processing", latest version tryed on CSGO server

[dparkinson101]

Turns out this was a problem unique to my setup, I checked the MariaDB while processes were running and found that it somehow mismatched server ids for certain servers mid-process and lock itself up, cannot replicate this issue on my new setup on Ubuntu so I'm marking this up to a weird error that only occurs on CentOS 8 which somehow happened during an upgrade.

To those above that are saying they are experiencing similar issues I would recommend a reinstall of the panel on an Ubuntu instance and then re-adding your instance of wings as a node on that panel. Or you could try to correct the id missmatch manually in the DB not since I don't know what caused it I would say this is a sign that CentOS support is dead in the water (since it's already at EOL anyway I'm fine with this honestly) and to just jump ship to Ubuntu or another OS.

Or trying doing systemctl restart pteroq or php artisan queue:restart in order to restart the process queue / jump start the panel.

Anyway since I'm fine with this issue being solved I'm going to close this up as it seems not much is going to happen with it other than making people with a stuck pteroq process think it's an unsolved bug.

[ghost]

Thats interesting. My WINGS install was on ubuntu but I found that the pteroq service restart counter had been reached preventing the service from starting properly. Running the following commands fixed the issue for me.

systemctl reset-failed systemctl daemon-reload systemctl restart pteroq

The first time I ran those it just kept triggering the failed counter. I fixed that by changing the schedules status column from processing to NULL in the MySQL DB and re-ran the commands and it seemed to fix the issue. I havent seen this issue arise again after v1.9

[jchambo]

I'm also having this issue. I've done the following commands with no luck or any change to the status:

systemctl reset-failed systemctl daemon-reload systemctl restart pteroq

I'm afraid to touch anything in the database.

I can confirm that the "Schedules" are not running at the times I specify. However if I delete the Schedules, and re-create them, and click on the "Run Now" option, they run perfectly fine. The issue seems to be with the schedule itself. It just gets stuck in a "Processing" state when the schedule kicks it off.

Any idea why this is happening with just the schedules but not when manually clicking "Run Now" ?

The pteroq service appears fine as well?

EDIT: SOLVED: I believe I have solved this thanks to the help in the discord. Solution was going into nano /var/www/pterodactyl/.env and removing the password I had set in REDIS_PASSWORD= and setting it to null so it now reads: REDIS_PASSWORD=null then refreshing the config cache with cd /var/www/pterodactyl && php artisan config:clear and restarting the pteroq service with systemctl restart pteroq