HMI: SP100_X2_4_02_BP_1_04_Master YDL Issue

[LewisPye]

Experiencing issues with an SP100 during commissioning activities:

• Intermittent success on live bus start
• Issues with unit faulting throughout the night daily
• Water cooling instability

[LewisPye]

Below email dialogue explains the extent of the issue.

From: Matthew Scales︱Magnus <[mjs@magnusmarine.com](mailto:mjs@magnusmarine.com)> 
Sent: Thursday, 18 July 2024 11:46 pm
To: Chris Lynch-Blosse <[chris@attest.co.nz](mailto:chris@attest.co.nz)>; [ian@yachtingdevelopments.co.nz](mailto:ian@yachtingdevelopments.co.nz)
Cc: [GrahameJ@yachtingdevelopments.co.nz](mailto:GrahameJ@yachtingdevelopments.co.nz); [mark@attest.co.nz](mailto:mark@attest.co.nz); Lewis Pye︱Magnus <[lbp@magnusmarine.com](mailto:lbp@magnusmarine.com)>; Daniel Trenaman︱Magnus <[dmt@magnusmarine.com](mailto:dmt@magnusmarine.com)>
Subject: Re: YDL 1024 - Maguns Shorepower Unit.

Hi Chris and All,

Thanks for the email and your efforts to date. Today is my first day on holiday and so have CC’d both Daniel and Lewis into the email. Both are senior guys and can make decisions without me. For the next week I will continue to review and contact them for daily updates.

Firstly, it is our priority to get this resolved. I work hard to avoid these issues.

This past week I have been observing the dialogue between Daniel and Natt. With lack of progress, it’s been somewhat frustrating and feel we need below answers to move forward. What’s App is proving not to be ideal as key information is missed.

I have highlighted outstanding issues so we can determine plan forward.

Cooling issues
Very little information on what is the issue or current state. 
1.  Is the valve operational. Can you force the valve open and close by changing the PID temp control settings on the display.
The valve is operational. We have observed the control voltage to be varied between 3-10Vdc. Also, we have observed that the valve stem was moving.
Ove thing to mention is that the valve seems to be at fully opened position at around 30% load.
Anyway, we did not try to alter valve control from the display.
1.  Is the valve plumbed in on the water inlet?
The valve is plumbed as per the valve manual.
1.  Valve installed right orientation.
Correct.
1.  Photo of the valve cooling page so we can confirm outlet air temp against valve position.
Please see the attached.
1.  Has the fan speed controller been changed from Manual 15% back to Auto. This is in my commissioning report to avoid dust ingress during your time in the workshop.
Mark has changed it to Auto when he first started testing the unit.
1.  Is the front panel on?
The front panel is always on except when we’re testing with Daniel.

Power Interruption
Other than the system tripping on Sunday at 05:20 on AFE fault we have little information. Some questions to help us understand:-

1.  System was turned on December last year. Is this interruption new or been apparent for many months?
The interruption started once the boat’s in the water and has never stopped. 05th June was the earliest date of AFE fault seen from the event log.
1.  How often has this happened?
This happens once every night. I’ve been informed that the boat cannot get through the night without the issue.
1.  Does this happen daily and / or during peak loads?
Once a night with no abnormal electrical activity.
1.  Is the vessel in the water?
Yes.
1.  What supply are you connected to 1ph or 3ph?
3-phase supply.
1.  What is the load on the vessel during the event?
Usually, the loads are around 7-20 kVA.

Seamless transfer
From last night’s performance it feels like we have gone backwards. Last week there was significant progress with only a few trips during transfer, so far, we have only adjusted droop and freq set points.

Past week it is unclear what has changed if anything. But we have noticed sometimes the transfer happens with two generators online which is just not possible. 

First, we need to understand setup onboard.

1.  What are the generators, I believe they operate in isochronous and have local control when coming online?
They are Kohler 80kW, 80EFOZDJ. They’re operated by PGEN and there is no local control.
1.  When the converter is online how do you stop the generator, is this manual or automatic?
The generator is stopped manually through local or remote display.
1.  Transferring from shore power back to generator, is this successful?
This has been always successful.
1.  Assume generator synchronises when transferring from shore?
Correct.
1.  What is the sync strategy onboard?
The Kohler PGEN takes control of the generators. When transferring from Shore to Gen, after ‘gen start’ button is pressed, PGEN will starts both generators, keep the duty one running and stop the other one. Once the duty generator and Magnus unit has shared the load, the Magnus unit will be manually stopped.
1.  Has anything changed with the generator from last week, settings etc?
No.
1.  Is there a mode generator operates in during the transfer?
The generators are controlled by PGEN.
1.  What setup and generators were installed on YDL build 1017 as this worked perfectly?
We don’t have this information.
1.  The procedure during transfer must be to go onto one generator. We have noticed two generators online which should be prevented in the future. The generators are sized at 80kW each and the load must be below 50kW, the size of the shore power.
Refer no.5.

If live bus synchronisation with generators is not compatible, then clearly we need to start making alternative plans. It’s unclear why this would be the case as YDL 1017 worked perfectly. Possible solution is to put a contactor on converter output and operate similar setup to generators. The converter will synchronise to generator before coming online by closing contactor. We have demonstrated yesterday the converter achieves a 9V sync which is highly accurate for smooth transfer. What is the procedure in turning off the generator.
The generators are manually turned off after the shore power unit is synced to the bus. The operator will press ‘stop’ from local/remote generator display.

Daniel mentioned Power Quality onboard as we had experienced where harmonics onboard had exceeded 15% causing similar synchronisation issue. This was resolved by turning off the ships Air conditioning compressors which subsequently reduced harmonics below 5%. Worth checking if you have a meter available.
We have an oscilloscope to check the harmonics.

I do not feel confident more remote support will get the results YDL require. If you are going onboard I may ask for you to do a few things, however a special trip is not required. This would involve moving the voltage sending from secondary side of contactor to primiary so the converter reads output voltage beofre contactor closes. Also would be good to clear history log.
If this involves any change in wirings, please send us a mark up on drawings to minimized unwanted outcome.

We can be available to come and visit 1st week of August. What is the scheulde of the vessel?

Best regards

Matthew Scales 
Managing Director
uk.[+44(0) 1953 661111](tel:+441953661111)
nl. [+31(0) 78744 04 35](tel:+31787440435)
m.[+44 (0) 797 1829939](tel:+447971829939)
e: [mjs@magnusmarine.com](mailto:mjs@magnusmarine.com)

[LewisPye]

Investigating the SP100 change log reveals project 1901 is operating X2 4.02 BP 1.04

[LewisPye]

The below code extract is the progression of the converter entering converter 'ON' mode

public void ModeAUTO()
{
    System.Threading.Thread autoThread;

    autoThread = new Thread(() => AutoThreadExec());
    autoThread.IsBackground = true;
    autoThread.Start();
}

private void AutoThreadExec()
{
    int timeout;

    // Thread locking
    if (autoThreadLock != 0 || bypassThreadLock != 0)
    {
        return;
    }
    autoThreadLock = 1;

    // Check for Water Sensor Fault
    if (Globals.Tags.Water_Alarm_Latched.Value == true)
    {
        Globals.Tags.Bypass_seqIncom.SetTag();
        Globals.Tags.Bypass_statustext.Value = "Water Ingress Fault";
        Globals.Tags.Bypass_seqIncom.Value = true;

        Globals.Tags.Bypass_dialogText.Value = "Water Ingress Fault\nOK to Reset\nCancel to Abort";
        Globals.Confirm_Custom.Show();

        timeout = 50;
        Globals.Tags.Bypass_dialogCode.Value = 0;
        while (Globals.Tags.Bypass_dialogCode.Value == 0 && timeout > 0)
        {
            Thread.Sleep(200);
            timeout--;
        }
        Globals.Bypass.Show();

        // OK was selected to send Reset command
        if (Globals.Tags.Bypass_dialogCode.Value == 2)
        {
            Globals.Tags.Water_Alarm_Latched.Value = false;
            Globals.Tags.CAN_CMD_RESET.SetTag();
            Thread.Sleep(400);
            ModeOFF();
            autoThreadLock = 0;
            return;
        }
        // Cancel was selected or dialog timed-out - drop out of thread without changing mode
        else
        {
            ModeOFF();
            autoThreadLock = 0;
            return;
        }                

        if (Globals.Tags.Bypass_Mode.Value == 0)
        {
            Globals.Tags.Bypass_statustext.Value = "System is OFF";
        }

        if (Globals.Tags.Bypass_Mode.Value == 2)
        {
            Globals.Tags.Bypass_statustext.Value = "CONV Mode Enabled";
        }

        autoThreadLock = 0;
        return;
    }            

    // We are now in AUTO mode
    Globals.Tags.Bypass_Mode.Value = 2;

    // Start of thread work
    Globals.Tags.Bypass_statustext.Value = "Entering CONV Mode";

    // Open Bypass contactor
    Globals.Tags.Application_IO__QbDigOutput1.Value = false;
    Thread.Sleep(400);

    // Close Converter Output Contactor (Set Relay (NO)) (0)
    Globals.Tags.Application_IO__QbDigOutput0.SetTag();

    // Start the converter
    Globals.Tags.CAN_CMD_ON.SetTag();

    // Wait with timeout to check converter has started
    timeout = 50; // 10 Seconds (50 * 200 ms)
    while (Globals.Tags.SP100_Status.Value != 3 && timeout > 0)
    {    
        Thread.Sleep(200);
        timeout--;
    }

    // Check for converter start timeout
    if (timeout <= 0)
    {
        Globals.Tags.Bypass_seqIncom.SetTag();
        Globals.Tags.Bypass_statustext.Value = "Start Timeout";
        Globals.Tags.Bypass_seqIncom.Value = true;

        Globals.Tags.Bypass_dialogText.Value = "Start Timeout\nOK to Reset\nCancel to Abort";
        Globals.Confirm_Custom.Show();

        timeout = 50;
        Globals.Tags.Bypass_dialogCode.Value = 0;
        while (Globals.Tags.Bypass_dialogCode.Value == 0 && timeout > 0)
        {
            Thread.Sleep(200);
            timeout--;
        }
        Globals.Bypass.Show();

        // OK was selected to send Reset command
        if (Globals.Tags.Bypass_dialogCode.Value == 2)
        {
            Globals.Tags.CAN_CMD_RESET.SetTag();
            Thread.Sleep(400);
            ModeOFF();
            autoThreadLock = 0;
            return;
        }
        // Cancel was selected or dialog timed-out - drop out of thread without changing mode
        else
        {
            ModeOFF();
            autoThreadLock = 0;
            return;
        }
    }

    // Give the converter some time to ramp up
    // Thread.Sleep(2000);

    Globals.Tags.Bypass_statustext.Value = "CONV Mode Enabled";

    // Set the last mode tag (non-volatile for autostart function
    Globals.Tags.Bypass_lastMode.Value = 2;

    // Clear the sequence incomplete flag
    Globals.Tags.Bypass_seqIncom.ResetTag();

    autoThreadLock = 0;
}

Analysis of Converter Starting into ON State

Here’s a step-by-step analysis of the converter starting process, including potential reasons for intermittent failures during live bus starts:

1. Initialisation and Thread Locking:

   • The process begins by ensuring no other mode transitions (bypass or auto) are currently in progress through thread locking (autoThreadLock and bypassThreadLock).
   • If either lock is set, the function exits, preventing simultaneous mode transitions.

2. Water Ingress Fault Check:

   • Before proceeding, the system checks for water ingress faults (Globals.Tags.Water_Alarm_Latched.Value).
   • If a fault is detected, the sequence incomplete flag is set, and an error dialog is displayed.
   • The function waits for user input to either reset the fault or cancel the operation.

if (Globals.Tags.Water_Alarm_Latched.Value == true)
{
    // Handle water ingress fault
}

3. Entering AUTO Mode:
   • The mode is set to AUTO (converter mode), and the status text is updated to indicate the transition.
   • The function proceeds to open the bypass contactor and introduces a 400 ms delay to ensure the contactor has fully opened.

Globals.Tags.Bypass_Mode.Value = 2;
Globals.Tags.Bypass_statustext.Value = "Entering CONV Mode";

Globals.Tags.Application_IO__QbDigOutput1.Value = false;
Thread.Sleep(400);  // Ensure the contactor is open

4. Closing Converter Output Contactor:
   • The converter output contactor is closed immediately after the bypass contactor is opened.

Globals.Tags.Application_IO__QbDigOutput0.SetTag();

5. Starting the Converter:
   • The system issues the command to start the converter (Globals.Tags.CAN_CMD_ON.SetTag()).
   • A timeout mechanism (set to 10 seconds) is used to wait for the converter to stabilise, indicated by Globals.Tags.SP100_Status.Value being set to 3.

Globals.Tags.CAN_CMD_ON.SetTag();
timeout = 50; // 10 seconds (50 * 200 ms)
while (Globals.Tags.SP100_Status.Value != 3 && timeout > 0)
{
    Thread.Sleep(200);  // Wait 200 ms
    timeout--;          // Decrement the timeout counter
}

6. Handling Start Timeout:
   • If the converter does not stabilise within the timeout period, the sequence incomplete flag is set, and an error dialog is displayed.
   • The function waits for user input to either reset the fault or cancel the operation.

if (timeout <= 0)
{
    // Handle start timeout
}

7. Finalising the Mode Transition:
   • If the converter starts successfully, the status text is updated, the last mode tag is set for auto-start functionality, and the sequence incomplete flag is cleared.

Globals.Tags.Bypass_statustext.Value = "CONV Mode Enabled";
Globals.Tags.Bypass_lastMode.Value = 2;
Globals.Tags.Bypass_seqIncom.ResetTag();

Code Review for Boundaries and Checks

• Voltage and Frequency Check:

  • Ensure ShoreSupplyOK() accurately checks the shore supply conditions.

  private bool ShoreSupplyOK()
  {
      if (Globals.Tags.Single_Phase_IP.Value == false)
      {
          if (Globals.Tags.CAN_IP_STATUS_FREQ.Value > 4800 && Globals.Tags.CAN_IP_STATUS_FREQ.Value < 5200)
          {
              if (Globals.Tags.CAN_IP_VSTAT_L1.Value > 380 && Globals.Tags.CAN_IP_VSTAT_L1.Value < 425)
              {
                  if (Globals.Tags.CAN_IP_VSTAT_L2.Value > 380 && Globals.Tags.CAN_IP_VSTAT_L2.Value < 425)
                  {
                      if (Globals.Tags.CAN_IP_VSTAT_L3.Value > 380 && Globals.Tags.CAN_IP_VSTAT_L3.Value < 425)
                      {
                          return true;
                      }
                  }
              }
          }
      }
      return false;
  }

• External Bus Live Check:

  • Ensure isExtBusLive() checks for minimal residual voltage.

  private bool isExtBusLive()
  {
      if (Globals.Tags.CAN_AUXBUS_VSTAT_L1.Value > 10 || Globals.Tags.CAN_AUXBUS_VSTAT_L2.Value > 10 || Globals.Tags.CAN_AUXBUS_VSTAT_L3.Value > 10)
      {
          return true;
      }
      return false;
  }

[LewisPye]

Looking plausible that the converter could be intermittently failing on a rising bus as this is connected simultaneously with the closure of the converter contactor.

[LewisPye]

Code preparation incase of deployment request - This will slot into the relevant codebase.

   // Set mode to AUTO
    Globals.Tags.Bypass_Mode.Value = 2;
    Globals.Tags.Bypass_statustext.Value = "Entering CONV Mode";

    // Open Bypass contactor
    Globals.Tags.Application_IO__QbDigOutput1.Value = false;

    // I recommend we increase the below timer from 400ms to 1000ms... A contactor can easily take 100ms to close.
    Thread.Sleep(1000);  // Ensure the bypass contactor is open

    // Close Converter Output Contactor
    Globals.Tags.Application_IO__QbDigOutput0.SetTag();
    // Additional delay to ensure the converter contactor is closed.. 2 seconds... possibly more. Slow & steady wins the race!
    Thread.Sleep(2000); 

    // Start the converter
    Globals.Tags.CAN_CMD_ON.SetTag();
    timeout = 50; // 10 seconds (50 * 200 ms)
    while (Globals.Tags.SP100_Status.Value != 3 && timeout > 0)
    {    
        Thread.Sleep(200);  // Wait 200 ms
        timeout--;          // Decrement the timeout counter
    }

[LewisPye]

Located the HMI part No.: M:\Engineering\Parts\Parts 1A\1A-800 Project Software\1A-800-106\HMI Software Master\SP100_X2_4_02_BP_1_05_20220803_Master_Remote

[LewisPye]

Hi Lewis,

I have followed the procedure, and the followings are the result.

At step 3, closing the shore power output MCB, once the MCB was closed, the shore power unit instantly shared the load with generator.
The generator was running solely around 30kVA then became 20kVA with another 10kVA on the shore power unit.
This was without pressing ‘start’ on the display.
I didn’t expect the converter sharing load without starting.

After that I followed to step 4, which is pressing ‘start’, once the button was pressed, the status changed to ‘sync’ for 2-3 seconds and then went to ‘fault’.
The fault was ‘IGBT Error’.
Even with faulty status, the shore power unit was still sharing load with the generator.
This was quite opposite to what mentioned in 3.d.i.

I have tested this twice and received same results.

[LewisPye]

Modified code and have exported to location: M:\Engineering\Parts\Parts 1A\1A-800 Project Software\1A-800-106\HMI Software Master\SP100_X2_4_02_BP_1_06_20240809_Master_Remote Export

[LewisPye]

Further increased time delay of starting convert to 5000ms

    /* We are now in AUTO mode */
            Globals.Tags.Bypass_Mode.Value = 2;

            /* Start of thread work */
            Globals.Tags.Bypass_statustext.Value = "Entering CONV Mode";

            /* Open Bypass contactor */
            //  Globals.Tags.Application_GVL_S1_M2_ST2328_Out1.Value = false;
            Globals.Tags.Application_IO__QbDigOutput1.Value = false;
            Thread.Sleep(1000);

            /* Close Converter Output Contactor (Set Relay (NO)) (0) */
            //  Globals.Tags.Application_GVL_S1_M2_ST2328_Out0.SetTag();
            Globals.Tags.Application_IO__QbDigOutput0.SetTag();

            Thread.Sleep(5000);

            /* Start the converter */
            Globals.Tags.CAN_CMD_ON.SetTag();
            /* Wait with timeout to check converter has started */
            timeout = 50; // 10 Seconds. 
            while (Globals.Tags.SP100_Status.Value != 3 && timeout > 0)
            {   
                Thread.Sleep(200);
                timeout--;
            }

[LewisPye]

<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">

Hi Matt,

 

1. PQ Status: What is the PQ status of the main bus? From previous correspondence I’ve witnessed high KVARs and clearly a distorted bus will produce challenges. Would like to know what we are dealing with.
2. Calibration: As we know it’s important to be accurately calibrated. Has this been done?
3. PLL Status: It’s important to gain a baseline understanding of the PLL lock status at it’s current configuration so we can monitor for improvements. With the converter offline and not connected to the bus, the PLL lock controller is idle. With the converter in a live bus scenario where voltage is detected on the converter output, the PLL lock controller will become active. Note: As soon as the converter status transitions to online, the PLL lock controller function is no longer called.
4. EEPROMS: In the former FW, unfortunately PLL controller gains were less accessible from EEPROM level. Some recommendations are below:
1. 0x8126 LSYNC_PI_VD_TIME – This is the time interval of the PI controller. Start by increasing this, perhaps double and then treble the value to monitor effects. You will soon gain an understanding of it’s effects and will soon realise if you’re heading in the right direction based on the baseline PLL recording you earlier taken.
2. 0x80A0 PLL_LOCK_THRES – Although we discussed earlier that the PLL lock threshold is a key stakeholder in deciding if the converter is in sync for a traditional sync, in the later versions of FW this value is included within the core PLL lock function and will assist in the PLL target decision making internally; essentially the aim is to attempt for a tighter lock. I recommend decreasing this value in orders of 10 down to 100 if needed.
3. 0x811E VSEN_LPF_ALPHA – This value comes hand in hand with the PQ status… If we are dealing with a noisy bus then we could look to adjust the low pass filter. Essentially the lower the number, the less filtering applied. You may want a more RAW signal or more filtered based on the PQ.
5. Software: I’ve sent the latest SW package via we transfer.
1. Disabled the customer MODBUS controller as default.
2. Implemented time delay of 5000ms before converter will start after the converter contactor closes and also implement a 1000ms time delay to allow the bypass contactor to successful open before progressing.
3. 
4. Be cautious when uploading. You should only need to update the HMI. Ensure you have noted the IP address of the screen before uploading the new SW incase this needs reconfiguring. Place the files on a USB stick. When it asks would you like to back up existing project, press YES!
6. Event log: Perhaps the event log needs resetting from the workbench using the R/W parameter tab.

 

Kind Regards Lewis Pye Technical Manager

Magnus Marine M: +44 7500 396823 P: +44(0) 1953 661111 E: lbp@magnusmarine.com

Hi Matt,

1. PQ Status: What is the PQ status of the main bus? From previous correspondence I’ve witnessed high KVARs and clearly a distorted bus will produce challenges. Would like to know what we are dealing with.
2. Calibration: As we know it’s important to be accurately calibrated. Has this been done?
3. PLL Status: It’s important to gain a baseline understanding of the PLL lock status at it’s current configuration so we can monitor for improvements. With the converter offline and not connected to the bus, the PLL lock controller is idle. With the converter in a live bus scenario where voltage is detected on the converter output, the PLL lock controller will become active. Note: As soon as the converter status transitions to online, the PLL lock controller function is no longer called.
4. EEPROMS: In the former FW, unfortunately PLL controller gains were less accessible from EEPROM level. Some recommendations are below: a. 0x8126 LSYNC_PI_VD_TIME – This is the time interval of the PI controller. Start by increasing this, perhaps double and then treble the value to monitor effects. You will soon gain an understanding of it’s effects and will soon realise if you’re heading in the right direction based on the baseline PLL recording you earlier taken. b. 0x80A0 PLL_LOCK_THRES – Although we discussed earlier that the PLL lock threshold is a key stakeholder in deciding if the converter is in sync for a traditional sync, in the later versions of FW this value is included within the core PLL lock function and will assist in the PLL target decision making internally; essentially the aim is to attempt for a tighter lock. I recommend decreasing this value in orders of 10 down to 100 if needed. c. 0x811E VSEN_LPF_ALPHA – This value comes hand in hand with the PQ status… If we are dealing with a noisy bus then we could look to adjust the low pass filter. Essentially the lower the number, the less filtering applied. You may want a more RAW signal or more filtered based on the PQ.
5. Software: I’ve sent the latest SW package via we transfer. a. Disabled the customer MODBUS controller as default. b. Implemented time delay of 5000ms before converter will start after the converter contactor closes and also implement a 1000ms time delay to allow the bypass contactor to successful open before progressing. c.
   d. Be cautious when uploading. You should only need to update the HMI. Ensure you have noted the IP address of the screen before uploading the new SW incase this needs reconfiguring. Place the files on a USB stick. When it asks would you like to back up existing project, press YES!
6. Event log: Perhaps the event log needs resetting from the workbench using the R/W parameter tab.

Kind Regards

Lewis Pye Technical Manager Magnus Marine M: +44 7500 396823 P: +44(0) 1953 661111 E: lbp@magnusmarine.com