Uncaught SyntaxError: Got [ERASED] expected EOF when writing to storage

[mariusGundersen]

I'm getting the following error in my program:

Compacting...-
Uncaught SyntaxError: Got [ERASED] expected EOF
 at line 136 col 46 in .bootcde
  Storage.write('image', "", 0, IMAGE_SIZE);
                                             ^
in function "run" called from line 210 col 7 in .bootcde

As you can see I'm trying to create a large file in flash that I can stream an http response into it. I'm trying to allocate it upfront, but when I do the espruino outputs "Compacting...-" and it seems that it is trying to compact the bootcode that it is running. This results in an error.

Is there something I'm doing wrong here? I have tried to create the file upfront using the espruino cli, using espruino ./firmware.js --config SAVE_ON_SEND=1 --storage image:image.bin (where image.bin is a large empty file, just to take up the space).

[gfwilliams]

That's an odd one - and your firmware is definitely reasonably up to date (like 2v18 or later?) and what device are you doing this on?

If Espruino does compact code and re-lays it out, it should go through memory and re-point any functions in code to the new locations. Any chance you could come up with a really minimal test case that exhibits this and I can see if I'll debug it.

But you have two choices for now really:

• Mark the function that writes to the code with "ram" right at the start - that'll keep it in RAM and then the fact it moves around won't matter. Other functions (as long as they're not being executed at the time) should be fine.
• Rather than allocating the image file first, just ensure that Storage is completely erased (require("Storage").eraseAll()) and then write your code. Now the code will be at the beginning of flash so won't ever change position even when compacted

[mariusGundersen]

Running on v2.19.

When I run Storage.list() I get this:

Storage.list()
=[
  "image",
  ".bootcde"
 ]

so looks like the image ends up before the bootcode. When I try Storage.eraseAll() it also removes .bootcde, so now it won't run when it reboots, right?

[gfwilliams]

what device is it?

When I try Storage.eraseAll() it also removes .bootcde, so now it won't run when it reboots, right?

Correct. So erase, then upload your code to flash, then boot code will be first and will always stay first

[mariusGundersen]

Sorry, this is an espruino pico.

The entire code is here: https://github.com/mariusGundersen/edp-frame/blob/compress/firmware/firmware.js.

[gfwilliams]

Thanks - I ask because it might use a different variable type which might not have been relocated - for example I don't believe this happens on Bangle.js because it uses FlashStrings for executing in place.

Ok, so I can reproduce by pasting this into the repl:

require("Storage").write("test",`This is some test test that should be reasonably long
This is some test test that should be reasonably long
This is some test test that should be reasonably long
This is some test test that should be reasonably long
This is some test test that should be reasonably long
This is some test test that should be reasonably long
This is some test test that should be reasonably long
This is some test test that should be reasonably long`);
require("Storage").write(".bootcde",`
print("Writing new file");
// write enough that we will have to compact
require("Storage").write("test", "", 0, require("Storage").getStats().totalBytes-300);
print("Finished with success");
`);
load();

When a file gets moved, jsvUpdateMemoryAddress gets called which scans all variables and changes their addresses.

But unfortunately I believe the boot code has been loaded as a 'nativestring' which is a specific pointer to an area of memory, and that pointer gets loaded into the iterator and isn't updated.

About the only solution I can see is to have Espruino reload the pointer from the variable after each function call it makes?

[mariusGundersen]

Is there a way to prevent compression from running again? It already tried to compress when uploading the file, so there really isn't anything to gain from compressing again. And there is enough room for the file.

[gfwilliams]

I'm afraid not - but it shouldn't compress if there is enough room for the file, so it obviously thinks there isn't.

What does require("Storage").getStats() say vs how much you want to allocate?

I should add that right now it tries to allocate files on the page boundary, and it's possible that is causing issues here as on the Pico the pages are pretty huge. But again a really minimal test case like I posted above would help me to narrow that down.

[gfwilliams]

Should be fixed - I believe the issue with compaction is https://github.com/espruino/Espruino/issues/2232 though.

[mariusGundersen]

I'm guessing there is a reason for doing it differently in bangle vs pico?

Also, I'm wondering why it does the compression when the file already exists and didn't change size. I can understand when it's a new file, but when I'm writing to an existing allocated file, why would it try to change the file layout?

[mariusGundersen]

Looking through the source I see that it only calls createFile when writing to offset=0, so I can try to write to offset 1 and then ignore the first byte when reading back again, that should prevent it from doing anything with the file

[gfwilliams]

I'm guessing there is a reason for doing it differently in bangle vs pico?

Yes - the Bangle has external flash that can't be memory mapped.

why it does the compression when the file already exists

It shouldn't - but doing require("Storage").write("test", "", 0, 100) forces a new file to be written and the old one to be erased. But if I had a snippet of code that exhibits the problem without me having to wire up an ESP8266 it'd be easy for me to look into why there were issues.

I see that it only calls createFile when writing to offset=0

Yes - I'll update the docs to make this clearer, it only mentioned it in the code example.

There's definitely an argument that we should use nonzero file length to signify that we should write a new file, rather than offset=0, but I'm not sure whether that'll break any existing code