Farmerbot - Fujitsu serwer issues

[JerzyJasiewicz]

Hi,

I have 2 farms with Fujitsu RX600 S6 servers. The first farm has 1 Fujitsu, 3 Dell R820 and 1 Lenovo micro PC (running farmerbot), the second farm has 2 Fujitsu and 1 Lenovo micro PC (with FB). Both farms have problems with Fujitsu - the main problem is that when farmerbot sends the "power down" command, the server does not shut down. It freezes and does not respond to any command (for example, alt+f2). The server never shuts down, I have to restart it manually (reset with power button).

This issue is on both farms. For now i have updated BIOS and iRMC. Motherboard have option to use WoL(authorized by bios) and ACPI (authorized by OS) - i have tired on both options, nothing changed. I have tested on diffirent LANS and routers (2 locations) - still occures.

I use 1 patchcord for zos/iRMC

Farms IDs : 5 and 863

config.md is now configured to not trun off nodes.

[muhamadazmy]

@xmonader I don't think this is a farmerbot issue if the farmer bot set the power target to down the node should shutdown. The zos info screen (although not pretty) looks like it was trying to shutdown. but I don't get why it is frozen according to @JerzyJasiewicz

I will look into the node logs and try to figure out what is going on

[muhamadazmy]

I noticed few issues regarding node 863:

• I can't see any logs from the node for the past 7 days
• The node is offline while the power target is UP. it should be online but it's not. (have you physically turn it off?)
• Freezing on powerdown is weird so i am wondering:
  • How did you boot the node? are u using a usb image from bootstrap ?
• It also look like the node is low on funds. (there is a fix to make node re-activate their accounts when the funds are low, but that did not reach production yet)

It's weird that the node logs ends arbitrary on the 13th of April.

[JerzyJasiewicz]

Nodes are up all the time, i turned them up by power button, additionaly i have set up in config.md to never turn off. In polkadot account there is about 4,8k of tft.

Nodes using USB stick to boot up, i have deployed farmerbot in may if i good remember, but there were issues so (described above) so i have set up "never turn off" on FB and since this time they work normally

[JerzyJasiewicz]

Yesterday i have updated bios and iRMC do i tired to setup farmerbot again but the issue still occure

[muhamadazmy]

Ah sorry, i checked the wrong node id. I will take another look. Seems i need more coffee haha

[JerzyJasiewicz]

Okay :), same issue i have on farm 5, there is one Fujitsu

[muhamadazmy]

Node 896 logs seems to be okay (also node is online and reachable). It's kinda hard to spot the time where it froze trying to shutdown (unless u know exact day to inspect). I am trying to find the logs to find out if it did say anything during the shutdown to explain why it froze.

Same for node 4634.

Do you remember any specific dates when u got this problem ?

[JerzyJasiewicz]

It was yesterday about 19:30 gmt+1

[brandonpille]

I notice you set the never_shutdown to true for all nodes. This means these nodes will never be shutdown...

[JerzyJasiewicz]

Ofc i did because if there wasn't "never shutdown" my nodes were freezing due to this issue. So for now farmerbot is working but there aren't anything to shutdown

[JerzyJasiewicz]

Hello, Any new ideas?

[scottyeager]

Seems same errors seen in https://github.com/threefoldtech/zos/issues/1968

[brandonpille]

@muhamadazmy any progress?

[JerzyJasiewicz]

Nothing changed for now. This issues touches all Fujitsu serwers that i know (6 nodes)

[muhamadazmy]

I still can't think of anything that can cause this except kernel compatibility issue with this hardware. I am not sure if a BIOS update could help but you can try that

[xmonader]

Closing, please reopen if still an issue