search

mtconnect / cppagent

C++ Agent toolkit - Pre-built binaries, visit: https://github.com/mtconnect/cppagent/releases Docker images available at https://hub.docker.com/repositories/mtconnect
http://mtcup.org/
Apache License 2.0
143 stars 91 forks source link

Agent is crashed when receiving command from client???? #487

Closed hulinning2 closed 2 months ago

hulinning2 commented 2 months ago

Hi @wsobel I use the latest public release MTConnect Agent, version 2.4.0.1, for Win32 on Windows 7 x64. I notice that MTConnect agent will be crashed without any error even in debug mode (see log file) in the following test sequences: 1/ Stop both MTConnect Agent and Adapter 2/Start MTConnect Agent (fully started) 3/ Start MTConnect Adatper OK so far no issue 4/Send a current command to the MTConnect Agent from MS Edge web browser: http://localhost:5000/current ==> Agent is crashed right after it was receiving a GET current command from browser.

I restart MTConnect Agent while MTConnect Adatper is STILL running, and it is working as normal after that However, if I close MTConnect Adapter and run it again, the same error will be happening again. I tested with MTConnect spec 1.3 and 2.0 and I got same error.

Replace the MTConnect Agent with version 1.6.0.7 and repeat the above test and I got no issue.

Here is the log file from Agent 2.4.0.1:

agent-error.log Device and Agent Config file.zip

Thanks

wsobel commented 2 months ago

Can you provide the adapter output when the adapter first connects to the agent?

Thanks W

wsobel commented 2 months ago

What I mean is the adapter output to the agent.

hulinning2 commented 2 months ago

Here is the output from adapter. All tags should be reset at first connection. 2024-09-11T15:38:27.0579270Z|avail|UNAVAILABLE|fmode|UNAVAILABLE|S1ovr|UNAVAILABLE|S1Mode|UNAVAILABLE|S1ChuckState|UNAVAILABLE|S2ovr|UNAVAILABLE|S2Mode|UNAVAILABLE|S2ChuckState|UNAVAILABLE|C3Mode|UNAVAILABLE|S6ovr|UNAVAILABLE|S6Mode|UNAVAILABLE|B1Mode|UNAVAILABLE|estop|UNAVAILABLE|pmode|UNAVAILABLE|pprogram|UNAVAILABLE|pexecution|UNAVAILABLE|pFovr|UNAVAILABLE|ppartcount|UNAVAILABLE|p1CommonVariable|UNAVAILABLE|p1MacManPanelHistory|UNAVAILABLE|p1MachineOperationPanelOutputDryRun|UNAVAILABLE|p1MachineOperationPanelOutputMachineLock|UNAVAILABLE|PlcMonitorIO_1|UNAVAILABLE|PlcMonitorIO_2|UNAVAILABLE|PlcMonitorIO_3|UNAVAILABLE|PlcMonitorIO_4|UNAVAILABLE|PlcMonitorIO_5|UNAVAILABLE|PlcMonitorIO_6|UNAVAILABLE|PlcMonitorIO_7|UNAVAILABLE|PlcMonitorIO_8|UNAVAILABLE|PlcMonitorIO_9|UNAVAILABLE|PlcMonitorIO_10|UNAVAILABLE|p1ProgramHeader|UNAVAILABLE|p1PalletID|UNAVAILABLE|p1block|UNAVAILABLE|p1line|UNAVAILABLE|p1CurrentTool|UNAVAILABLE|p1ToolAssetId|UNAVAILABLE|p1BlockNumber|UNAVAILABLE|S1speed|UNAVAILABLE|S1cmd|UNAVAILABLE|S1load|UNAVAILABLE|S1SurfaceSpeedA|UNAVAILABLE|S2speed|UNAVAILABLE|S2cmd|UNAVAILABLE|S2SurfaceSpeedA|UNAVAILABLE|C3actm|UNAVAILABLE|C3actw|UNAVAILABLE|C3load|UNAVAILABLE|S6speed|UNAVAILABLE|S6cmd|UNAVAILABLE|S6load|UNAVAILABLE|B1actw|UNAVAILABLE|B1load|UNAVAILABLE|X1actm|UNAVAILABLE|X1actw|UNAVAILABLE|X1load|UNAVAILABLE|Z1actm|UNAVAILABLE|Z1actw|UNAVAILABLE|Z1load|UNAVAILABLE|Z4actm|UNAVAILABLE|Z4actw|UNAVAILABLE|Z4load|UNAVAILABLE|YI1actm|UNAVAILABLE|YI1actw|UNAVAILABLE|YS1load|UNAVAILABLE|pTotalOperatingTime|UNAVAILABLE|pTotalRunningTime|UNAVAILABLE|pTotalCuttingTime|UNAVAILABLE|pTotalSpindleRunTime|UNAVAILABLE|pOperatingTime|UNAVAILABLE|pRunningTime|UNAVAILABLE|pCuttingTime|UNAVAILABLE|pSpindleRunTime|UNAVAILABLE|p1Fact|UNAVAILABLE|p1Fcmd|UNAVAILABLE|p1LPathPos|UNAVAILABLE|p1Fract|UNAVAILABLE|p1Frcmd|UNAVAILABLE

hulinning2 commented 2 months ago

all messages sent to agent for a full scan cycle: 2024-09-11T15:55:10.7973533Z|avail|UNAVAILABLE|fmode|UNAVAILABLE|S1ovr|UNAVAILABLE|S1Mode|UNAVAILABLE|S1ChuckState|UNAVAILABLE|S2ovr|UNAVAILABLE|S2Mode|UNAVAILABLE|S2ChuckState|UNAVAILABLE|C3Mode|UNAVAILABLE|S6ovr|UNAVAILABLE|S6Mode|UNAVAILABLE|B1Mode|UNAVAILABLE|estop|UNAVAILABLE|pmode|UNAVAILABLE|pprogram|UNAVAILABLE|pexecution|UNAVAILABLE|pFovr|UNAVAILABLE|ppartcount|UNAVAILABLE|p1CommonVariable|UNAVAILABLE|p1MacManPanelHistory|UNAVAILABLE|p1MachineOperationPanelOutputDryRun|UNAVAILABLE|p1MachineOperationPanelOutputMachineLock|UNAVAILABLE|PlcMonitorIO_1|UNAVAILABLE|PlcMonitorIO_2|UNAVAILABLE|PlcMonitorIO_3|UNAVAILABLE|PlcMonitorIO_4|UNAVAILABLE|PlcMonitorIO_5|UNAVAILABLE|PlcMonitorIO_6|UNAVAILABLE|PlcMonitorIO_7|UNAVAILABLE|PlcMonitorIO_8|UNAVAILABLE|PlcMonitorIO_9|UNAVAILABLE|PlcMonitorIO_10|UNAVAILABLE|p1ProgramHeader|UNAVAILABLE|p1PalletID|UNAVAILABLE|p1block|UNAVAILABLE|p1line|UNAVAILABLE|p1CurrentTool|UNAVAILABLE|p1ToolAssetId|UNAVAILABLE|p1BlockNumber|UNAVAILABLE|S1speed|UNAVAILABLE|S1cmd|UNAVAILABLE|S1load|UNAVAILABLE|S1SurfaceSpeedA|UNAVAILABLE|S2speed|UNAVAILABLE|S2cmd|UNAVAILABLE|S2SurfaceSpeedA|UNAVAILABLE|C3actm|UNAVAILABLE|C3actw|UNAVAILABLE|C3load|UNAVAILABLE|S6speed|UNAVAILABLE|S6cmd|UNAVAILABLE|S6load|UNAVAILABLE|B1actw|UNAVAILABLE|B1load|UNAVAILABLE|X1actm|UNAVAILABLE|X1actw|UNAVAILABLE|X1load|UNAVAILABLE|Z1actm|UNAVAILABLE|Z1actw|UNAVAILABLE|Z1load|UNAVAILABLE|Z4actm|UNAVAILABLE|Z4actw|UNAVAILABLE|Z4load|UNAVAILABLE|YI1actm|UNAVAILABLE|YI1actw|UNAVAILABLE|YS1load|UNAVAILABLE|pTotalOperatingTime|UNAVAILABLE|pTotalRunningTime|UNAVAILABLE|pTotalCuttingTime|UNAVAILABLE|pTotalSpindleRunTime|UNAVAILABLE|pOperatingTime|UNAVAILABLE|pRunningTime|UNAVAILABLE|pCuttingTime|UNAVAILABLE|pSpindleRunTime|UNAVAILABLE|p1Fact|UNAVAILABLE|p1Fcmd|UNAVAILABLE|p1LPathPos|UNAVAILABLE|p1Fract|UNAVAILABLE|p1Frcmd|UNAVAILABLE 2024-09-11T15:55:10.7973533Z|UserMessage1||UNAVAILABLE 2024-09-11T15:55:10.7973533Z|system|UNAVAILABLE|||| 2024-09-11T15:55:10.7973533Z|CoolantSystem1_cond|UNAVAILABLE|||| 2024-09-11T15:55:10.7973533Z|ElectricSystem1_cond|UNAVAILABLE|||| 2024-09-11T15:55:10.7973533Z|HydraulicSystem1_cond|UNAVAILABLE|||| 2024-09-11T15:55:10.7973533Z|LubricationSystem1_cond|UNAVAILABLE|||| 2024-09-11T15:55:10.7973533Z|PneumaticSystem1_cond|UNAVAILABLE|||| adapterVersion:UNAVAILABLE adapterVersion:UNAVAILABLE *mtconnectVersion:UNAVAILABLE 2024-09-11T15:55:10.7992202Z|@REMOVE_ALL_ASSETS@|CuttingTool

wsobel commented 2 months ago

Any additional data before the current? Just want to reproduce

hulinning2 commented 2 months ago

Here is all data putty.log

wsobel commented 2 months ago

It may be the adapter version. I'll check this out next week when I'm back from IMTS.

hulinning2 commented 2 months ago

Will, According to Windows Events log, agent.exe has an access code violation. Does that ring any bell?

Thanks

Log Name: Application Source: Application Error Date: 9/17/2024 2:35:57 PM Event ID: 1000 Task Category: (100) Level: Error Keywords: Classic User: N/A Computer: DPX-LHUYNH.okumant.okuma.com Description: Faulting application name: agent.exe, version: 2.4.0.1, time stamp: 0x66aa3fae Faulting module name: agent.exe, version: 2.4.0.1, time stamp: 0x66aa3fae Exception code: 0xc0000005 Fault offset: 0x0036f6ba Faulting process id: 0x9d64 Faulting application start time: 0x01db09304cd40f41 Faulting application path: D:\Program files\Okuma\Okuma MT Connect Adapter\agent.exe Faulting module path: D:\Program files\Okuma\Okuma MT Connect Adapter\agent.exe Report Id: 9fb65c6d-b74c-4149-bf2f-ed9d7289f99c Faulting package full name: Faulting package-relative application ID: Event Xml:

1000 0 2 100 0 0x80000000000000 85897 Application DPX-LHUYNH.okumant.okuma.com agent.exe 2.4.0.1 66aa3fae agent.exe 2.4.0.1 66aa3fae c0000005 0036f6ba 9d64 01db09304cd40f41 D:\Program files\Okuma\Okuma MT Connect Adapter\agent.exe D:\Program files\Okuma\Okuma MT Connect Adapter\agent.exe 9fb65c6d-b74c-4149-bf2f-ed9d7289f99c
wsobel commented 2 months ago

There is something in the XML generation that is causing an exception I am not catching. I need to reproduce. I’m working it this week.

wsobel commented 2 months ago

Having trouble reproducing on mac os with debugger. Setting up windows environment to test. When you use the same stream from putty, can you reproduce the crash?

wsobel commented 2 months ago

I'm having trouble reproducing. I am running the pre-build binary from the site with the adapter simulator replaying the putty log and no luck.

How are you invoking the current? Is it from a browser? If so, it may be one of the other file assets that could be causing an issue. Just want to make sure I can replicate the issue.

wsobel commented 2 months ago

Also, is this the x86 or x64 version of the agent?

hulinning2 commented 2 months ago

Will, I cannot reproduce the crash by just sending the putty stream to it. The sequence is important: 1/Agent and Adapter stops, and no browser that currently has current commands with agent such as http://localhost:5000/current 2/Start Agent 3/ Start Adapter and send data over 4/check http://localhost:5000 on MS Edge browser

It is x86 version of the agent. I turn off the Assets and agent still crash Thanks

wsobel commented 2 months ago

By file assets I meant the schema and xslt static files you have in the directory, including the favicon. Can you please upload them as well? Since you are using a browser, all the related files will be requested.

It could have to do with a related file request.

Thanks, W

hulinning2 commented 2 months ago

Will, Here are the files. Thanks

Okuma MT Connect Adapter - For Will.zip

wsobel commented 2 months ago

I have tried to recreate this problem with the exact sequence you specified and have had no luck reproducing it.

I don't know if it's related to the hardware/software you're running. What type of computer is this running on, and what version of Windows is it running on? I'm running on a Windows 11 VM and may need to buy a Windows box to test or use a cloud server.

Any additional info would be helpful, so far, striking out.

hulinning2 commented 2 months ago

I test on win10 x64.

hulinning2 commented 2 months ago

@wsobel I think I can narrow down to where agent is crashed. I think there are some issues with the monitoring files thread routines when adapter is running and changes the devices.xml/agent.cfg files. As a result, agent will detect and perform its action. After connecting and sending ping pong to adapter, I send a current command from browser and agent is crashed. I repeat the same test and adapter does not change the configuration files and agent works OK.

Here is the debug output files containing 2 scenarios described above.

Thanks AgentOutputFiles.zip

wsobel commented 2 months ago

Ok, ahhhhh. This is a very different sequence.

  1. Agent is running
  2. Device.xml and/or agent.cfg changes
  3. Agent reloads Devices.xml and/or agent.cfg and does a warm start
  4. Adapter disconnects
  5. Agent warm start completes
  6. Adapter reconnects
  7. Current causes the agent to crash.

Is that the sequence?

Do you know if it makes a difference if only the agnet.cfg or the Devices.xml file changes? will narrow down the testing.

Thanks,

wsobel commented 2 months ago

The log files you sent shows only the Devices.xml file is dectected changing. The agent diffs the Devices.xml and found no changes. Looks like the same file was reloaded. This means that the restart process changed the Device.xml file and broke something, from what I can tell.

I will try the same sequence of operations.

  1. Agent is running
  2. Device.xml changes
  3. Agent reloads Devices.xml and no chantges are detected
  4. Agent warm start completes
  5. The agent is stopped
  6. Agent restarts
  7. Adapter starts
  8. get /current causes crash

Is that the sequence?

Thanks,.

hulinning2 commented 2 months ago

Will, The adapter process is not restarted itself once it is running. My tcp thread might detect agent is disconnected and connected again. The main thing is agent detects file change and crash after receiving a current command.

Thanks

wsobel commented 2 months ago

Thanks for the clarification

hulinning2 commented 2 months ago

Will, Please test the case that agent.cfg and devices.xml could be changed altogether when MTConnect spec is changed. Adapter is not restarted.

hulinning2 commented 2 months ago

@wsobel Are you able to reproduce the error on your PC when devices.xml is reloaded?

Thanks

wsobel commented 2 months ago

I have reproduced the issue using the release build. I am not sure why it is failing, but I'm getting closer. The problem does not occur in the debug build. I will need to dig deeper, but I'm running into issues with the VS 2022 build system and conan. There is a bug in the VS 2022 vcvars that does not find the toolset for 14.3, causing the dependencies to fail.

I am working on solving that problem and looking into the crash. I need to do some more digging to get everything working. Should have an update soon.

wsobel commented 2 months ago

I have resolved the build issues with 2022 and reproduced them with VS 2022 in the RelWithDebInfo configuration. Unfortunately, all the details are optimized away, so I will try backing out optimizations until I can produce without them. I have at least located where the issue is occurring; now to find the cause.

wsobel commented 2 months ago

I have found the problem and have a fix in place. I have a PR open right now to address the issue. We should test the other device update workflows as well.