search

sparkfunX / Artemis_Global_Tracker

A global satellite tracker utilising the SparkFun Artemis module, Iridium 9603N satellite transceiver and u-blox ZOE-M8Q GNSS
Other
14 stars 10 forks source link

Examples stopped working after update to 2.1.0 #36

Closed razvandragomirescu closed 2 years ago

razvandragomirescu commented 3 years ago

The new examples compile fine but fail to start the modem. The output on the serial monitor is

Enabling the supercapacitor charger... Supercapacitors charged! Enabling 9603N power... Starting modem...

It appears to freeze there. I got no errors when compiling or uploading. Any way to debug this?

razvandragomirescu commented 3 years ago

The old examples still work fine (on the same hardware) with version 1.2.3, so it's not a hardware issue.

PaulZC commented 3 years ago

Hi @razvandragomirescu , Do you see the same behavior on all of the Iridium examples? Do the simple examples like Example7 work? Best wishes, Paul

PaulZC commented 3 years ago

Also, please make sure you have "RedBoard Artemis ATP" selected as the board type. ("Artemis Module" will not work.)

PaulZC commented 3 years ago

To enable debug messages for the Iridium serial communication, change the false in this line to true:

https://github.com/sparkfunX/Artemis_Global_Tracker/blob/8094e148241947c6812ac86424d9a6165f34f049/Software/examples/Example7_GetIMEI/Example7_GetIMEI.ino#L63

PaulZC commented 3 years ago

Hi @razvandragomirescu ,

If you have time, please do try uploading one of the new binaries onto your board. Details are here. That way we can rule out the way the code is being compiled.

If you run the Example7 binary, you should see:

Artemis Global Tracker
Example: Get IMEI

Please check that the Serial Monitor is set to 115200 Baud
and that the line ending is set to Newline.
Then click Send to start the example.

Enabling the supercapacitor charger...
Supercapacitors charged!
Enabling 9603N power...
Starting modem...
IMEI is 3004-------4090.
Putting the 9603N to sleep.
Disabling 9603N power...
Disabling the supercapacitor charger...
Done!

Running Example7 with DIAGNOSTICS set to true should produce:

Artemis Global Tracker
Example: Get IMEI

Please check that the Serial Monitor is set to 115200 Baud
and that the line ending is set to Newline.
Then click Send to start the example.

Enabling the supercapacitor charger...
Supercapacitors charged!
Enabling 9603N power...
Starting modem...
Calling internalBegin
Powering on modem...
>> AT

Waiting for response OK

<< AT

OK
>> ATE1

Waiting for response OK

<< ATE1

OK
>> AT&D0

Waiting for response OK

<< AT&D0

OK
>> AT&K0

Waiting for response OK

<< AT&K0

OK
Ring alerts are enabled.
>> AT+SBDMTA=1

Waiting for response OK

<< AT+SBDMTA=1

OK
>> AT+CGMR

Waiting for response OK

<< AT+CGMR

Call Processor Version: TA16005

Modem DSP Version: 1.7 svn: 2358

DBB Version: 0x0001 (ASIC)

RFA Version: 0x0007 (SRFA2)

NVM Version: KVS

Hardware Version: BOOT07d2/9603NrevDE/04/RAW0c

BOOT Version: TA16005 (rev exported)

OK
Firmware version is TA16005
MSSTM workaround is NOT enforced.
InternalBegin: success!
>> AT+CGSN

Waiting for response OK

<< AT+CGSN

3004-------4090

OK
IMEI is 3004-------4090.
Putting the 9603N to sleep.
Powering off modem...
Disabling 9603N power...
Disabling the supercapacitor charger...
Done!

Please also check that the red 9603N POWER LED turns on when you see the "Enabling 9603N power" appear.

If you want to debug further, you can try probing on the TX1 test point on the rear of the PCB. You should see the serial AT commands being sent to the modem at 19200 Baud (3.3V). The replies - if any - will appear on RX1.

Best wishes, Paul

razvandragomirescu commented 3 years ago

I have tried both on a new computer (Windows this time to rule out an issue with my Mac) and with the binaries (uploaded from Windows using the binary uploader). Same result, it freezes at "Starting modem...". The orange "charging" LED lights up briefly while the console says "Waiting for supercapacitors to charge", then the RED "9603N Power" LED lights up but the modem doesn't appear to respond.

Once again, this works fine with the older 1.2.3 firmware.

PaulZC commented 3 years ago

Hi @razvandragomirescu ,

Thank you for investigating this. Just to be clear, have you tried the Example7 binary? And does that freeze at the same place? (I'm trying to rule out an issue with re-starting Serial1 after the Artemis has been in deep sleep. Example7 does not use deep sleep.)

Thanks again, Paul

razvandragomirescu commented 3 years ago

Correct, it's Example 7 that I've tried, as per your suggestion. The behaviour is the same on both Mac and Windows, both when compiling from source and when loading the binary. Maybe my AGT board is faulty somehow (although that would not explain how/why it works with 1.2.3 but not with 2.1.0)? I only have one, so there's not much I can do about it.

razvandragomirescu commented 3 years ago

BTW, with DIAGNOSTICS=true, it stops at

Calling internalBegin Powering on modem...

I never get that "AT" command sent to the modem, it never displays that, it just freezes at "Powering on modem...".

PaulZC commented 3 years ago

Thanks again for the diagnostics.

Wow, OK. This is a head-scratcher...

Best guess, some badness is happening when the new code tries to enable the modem via the 'sleep' pin. Why? I did have to change the gnssON and gnssOFF functions in v2.0 because the power glitch issue I fixed months ago on the GNSS enable pin came back again. I wonder if this issue only appear on some pins on some chips. I will try and send you a new binary to try in the next day or two. (It means hacking the IridiumSBDi2c library as the control of the 'sleepPin' is embedded in there.)

Thank you for your patience.

razvandragomirescu commented 3 years ago

I've just installed and run the binary for the iridiumSleepTest, result is below:

Artemis Global Tracker Example: Get IMEI

Please check that the Serial Monitor is set to 115200 Baud and that the line ending is set to Newline. Then click Send to start the example.

Enabling the supercapacitor charger... Supercapacitors charged! Enabling 9603N power... Starting modem... Calling internalBegin custom configureSleepPin: sleepPin configured Powering on modem...

Still appears to freeze on "Powering on modem". Would it help if I mailed you the AGT unit I'm testing on (if you can't reproduce it on your devices)?

PaulZC commented 3 years ago

Hi @razvandragomirescu ,

(Wow - that was fast! I was just typing a message to you but you beat me to it!)

OK. That's very interesting. The code is executing the custom configureSleepPin because we see the custom configureSleepPin: sleepPin configured message.

https://github.com/sparkfun/SparkFun_IridiumSBD_I2C_Arduino_Library/blob/4652dc95da1e2f7425a4005bbe88dd29f3889591/src/IridiumSBD.cpp#L1106

Then we see the Powering on modem... message:

https://github.com/sparkfun/SparkFun_IridiumSBD_I2C_Arduino_Library/blob/4652dc95da1e2f7425a4005bbe88dd29f3889591/src/IridiumSBD.cpp#L1114

But it must be failing in the custom setSleepPin function - or just after it - because we do not see the custom setSleepPin: sleepPin set message.

I'm based in the UK. If you return your board to SF in Boulder, it will get to me but it will take a while. If I'm not trying your patience too much, can we try a couple more custom binaries first? I'd like to understand exactly where the code is failing. I'll upload another binary in a few minutes.

Best wishes, Paul

PaulZC commented 3 years ago

Updated files are there now... Thanks!

razvandragomirescu commented 3 years ago

Artemis Global Tracker Example: Get IMEI

Please check that the Serial Monitor is set to 115200 Baud and that the line ending is set to Newline. Then click Send to start the example.

Enabling the supercapacitor charger... Waiting for supercapacitors to charge... Waiting for supercapacitors to charge... Waiting for supercapacitors to charge... Waiting for supercapacitors to charge... Waiting for supercapacitors to charge... Waiting for supercapacitors to charge... Waiting for supercapacitors to charge... Waiting for supercapacitors to charge... Waiting for supercapacitors to charge... Waiting for supercapacitors to charge... Waiting for supercapacitors to charge... Supercapacitors charged! Enabling 9603N power... Starting modem... Calling internalBegin custom configureSleepPin: sleepPin configured Powering on modem... custom setSleepPin: sleepPin tristate_disable custom setSleepPin: sleepPin set HIGH

With the latest version.

PaulZC commented 3 years ago

Hi @razvandragomirescu ,

Thank you for trying the new code. It is clear that the test code is crashing when the tri-state pin is set to an output:

https://github.com/sparkfunX/Artemis_Global_Tracker/blob/df54d12fc1fb02c9f4aba7c07be63f8db4589229/Software/examples/AGT_iridiumSleep_Test/AGT_iridiumSleep_Test.ino#L107

The crash is actually happening during the 10ms delay after that line.

I have a board here which does the same thing on the gnssEN pin. But on my board the Iridium enable (sleepPin) works fine. On my board, I first saw the issue using v1.2 of the core. I changed the way I was driving the pin to correct it. Then with v2.1 it came back again and I had to add more changes to correct it (by making the output open-drain instead of push-pull). It does not seem to be any kind of short circuit or brown-out as I do not see a spike in the current draw when it happens. Nathan (@nseidle) thinks it might be an un-trapped interrupt - the pin could be generating an interrupt for which there is no service routine and the code crashes as a result. I will try and capture exactly what happens today using a hardware debug tool.

One more thing you could try is to make sure the Artemis bootloader is up to date. On the Uploader tool there is a second button labelled "Update Bootloader". Clicking that will update the bootloader to the latest version. Then you will need to upload the binary code again in the usual way. That might help.

We need to arrange to get a replacement board to you. How did you buy your AGT? Did you buy it directly from SparkFun?

Thanks again for all the time you've put into this - we really appreciate it.

Best wishes, Paul

razvandragomirescu commented 3 years ago

I've just tried to update the bootloader and reuploaded the test, it behaves exactly the same way as before. I've bought my AGT from Distrelec in Europe. Can you email me on the address in my Github profile please if you need details (or give me an address to send them to)? Thank you!

a-schneider-fmi commented 3 years ago

Since the update to version 2, I experience another issue which may or may not be related: the tracker seems not to wake from deep sleep after 2 messages have been sent. From the configuration tool, I see the following messages:

[...]
>>> Message sent! <<<
Clearing the MO buffer.
The number of messages in the MT queue is: 0
Putting the 9603N to sleep.
Getting ready to put the Apollo3 into deep sleep...
Disabling 9603N power...
Disabling the supercapacitor charger...
Powering down the GNSS...
Going into deep sleep until next WAKEINT (60 seconds).

And then it hangs. This happens with several TXINT settings (I tried 2 minutes, 5 minutes, 10 minutes).

I have tested example 12 to debug. It seems to work, at least it runs for more than half an hour. Do you have any suggestion how to investigate further?

PaulZC commented 3 years ago

Hi Andreas (@a-schneider-fmi ),

I don't think this is the same issue... Let's discuss it separately in #38.

Best wishes, Paul

PaulZC commented 3 years ago

Hi Andreas (@a-schneider-fmi ) & Razvan (@razvandragomirescu ),

I now suspect you are both seeing the same issue.

I confirmed this morning that holding the Serial1 RX pin low can cause the Artemis to hang - but only with v2.1 of the Apollo3 core. Details are here: https://github.com/sparkfun/Arduino_Apollo3/issues/423

When the Iridium modem is powered off - or possibly when it is first powered on - there is a good chance it holds the RX pin low causing the code hang.

I may be able to work around this by changing the code to make sure that Serial1 is only started once the modem is fully powered on. I will of course keep you both informed.

Sincere apologies for the inconvenience, Paul

razvandragomirescu commented 3 years ago

@PaulZC should I still return the unit to Sparkfun or is this solvable in software? I haven't shipped mine back yet, I was just about to call FEDEX.

PaulZC commented 3 years ago

Hi Razvan,

It may be possible to solve this with the AGT software. I'm not certain yet if I can correct this in the AGT code or whether it needs to be corrected within the Apollo3 core. I will try and let you know if a solution in the AGT code is possible as soon as possible. Perhaps you could delay calling FedEx until then?

Very best wishes, Paul

jerabaul29 commented 3 years ago

Following all these discussions with much interest, many thanks for your efforts making this whole environment better :) .

A small note to @razvandragomirescu : if you do not really NEED the core v2, have you considered using the core v1 for production for now? I am still sticking to the core v1 for my use, the only thing I miss is the bluetooth, all other stuff is working nicely. The core v1 has got really quite robust. I have deployed AGTs in the Arctic in February on drift ice based on the core v1, and my instruments are still working today, 7 months later. I had never had such success with deploying instrumentation in the Arctic before. So IMHO the core v1 is definitely production ready :) .

Still looking forward to a core v2 production ready though, many thanks @PaulZC for all your efforts :) .

PaulZC commented 3 years ago

Hi @razvandragomirescu / @a-schneider-fmi ,

Good news! I'm 99% confident I have a solution for this. I was having difficulty understanding why Example7 was failing for Razvan, but I do now understand why. I should be able to release updated examples late today / early tomorrow.

Thank you for your patience.

Very best wishes, Paul

PaulZC commented 3 years ago

Hi Razvan (@razvandragomirescu ),

I have updated the examples to include a fix for the Apollo3 Serial1 RX pin issue. Please give them a try. As usual, there are pre-compiled binaries you can upload. Or, you can compile the code yourself: please use v2.1.0 of the Apollo3 core; please update the IridiumSBDi2c library to v3.0.4.

Please let me know if this resolves your issue.

Very best wishes, Paul

razvandragomirescu commented 3 years ago

I'm afraid it doesn't fix it @PaulZC :

Artemis Global Tracker Example: Get IMEI

Please check that the Serial Monitor is set to 115200 Baud and that the line ending is set to Newline. Then click Send to start the example.

Enabling the supercapacitor charger... Supercapacitors charged! Enabling 9603N power... Starting modem... Calling internalBegin configureSleepPin: sleepPin configured Powering on modem... setSleepPin: sleepPin set HIGH custom IridiumSBD::beginSerialPort

Freezes at this point (I've only tried the Example 7, I can try the others if you want).

Razvan

PaulZC commented 3 years ago

Hi Razvan (@razvandragomirescu ),

Thank you for your persistence!

One last try... Please try Example7 again using v3.0.5 of the IridiumSBDi2c library. I have made a small change to the timing of beginSerialPort which I hope will help. I have updated the binaries to include the change - the examples themselves have not changed.

If this does not work then it really is time to call FedEx.

Very best wishes, Paul

razvandragomirescu commented 3 years ago

I'm afraid it's the same @PaulZC , I've just tried examples 7, 19 and 21, they all fail (freeze) at the same point. I'll ship the AGT back to Sparkfun on Monday next week, so if you have anything else you'd like me to try before then I'm happy to do it. Thank you!!

PaulZC commented 3 years ago

Thank you Razvan. I'm out of ideas... I need to get my hands on your board so I can figure out what is going wrong. Those examples all run OK on the board I have here. I do suspect it is the Serial1-hanging-when-RX-is-pulled-low issue but the changes I made should have fixed that. Thank you for your patience and understanding.

Let's leave this issue open until I have investigated your board. Very best wishes, Paul

razvandragomirescu commented 3 years ago

Unit shipped back to Sparkfun earlier today, I hope it helps @PaulZC .

PaulZC commented 3 years ago

Thank you Razvan. It will take a few days to reach me. I will update you on what I discover as soon as I have the board. Best wishes, Paul

PaulZC commented 2 years ago

Hi Razvan (@razvandragomirescu ), As we discussed yesterday, commit c6f3dbc does resolve this issue. Thank you again for all your help. Closing...! Very best wishes, Paul