Error: the JSON object must be str, not 'bytes'

[gudenau]

I followed the dirrections here to try and program a TinyFPGA BX that I just got and when I get to the upload step it errors with Error: the JSON object must be str, not 'bytes'. I guess this is some random Python issue?

What happened: Uploading the example project errors out with Error: the JSON object must be str, not 'bytes'

What should happen: The project should upload and the FPGA should run it.

Platform: uname -a : Linux DESKTOP-DEBIAN 4.9.0-6-amd64 #1 SMP Debian 4.9.88-1+deb9u1 (2018-05-07) x86_64 GNU/Linux

Steps to reproduce: (or what I did) sudo apt-get install python3-pip sudo pip3 install --upgrade --no-cache-dir apio==0.4.0b3 "tinyprog>=1.0.9" apio install system scons icestorm iverilog (since drivers does not exist) apio drivers --serial-enable tinyprog --update-bootloader Download atom-amd64.deb sudo dpkg -i atom-amd64.deb Install apio-ide in atom. git clone https://github.com/tinyfpga/TinyFPGA-BX.git cp TinyFPGA-BX/apio_template blink_project Open blink_project with atom. Upload. Error happens here.

[tinyfpga]

What's the exact version of python are you using?

Update tinyprog to 1.0.10:

sudo pip3 install --upgrade --no-cache-dir "tinyprog>=1.0.10"

Now run tinyprog -p bitstream_filename.bin to program the bitstream generated by APIO and paste any error message you get.

[gudenau]

Python 3.5.3, and would that be hardware.bin?

[tinyfpga]

Yeah, it think it’s hardware.bin.

On Sun, Jul 15, 2018 at 1:07 PM gudenau notifications@github.com wrote:

Python 3.5.3, and would that be hardware.bin?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tinyfpga/TinyFPGA-BX/issues/2#issuecomment-405114781, or mute the thread https://github.com/notifications/unsubscribe-auth/AcnVg2ug-PtmfM-Vzcvh3q_7QF2eYqPVks5uG6EMgaJpZM4VP-_N .

[gudenau]

    TinyProg CLI
    ------------
    Using device id 1d50:6130
    Only one board with active bootloader, using it.
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/serial/serialposix.py", line 265, in open
    self.fd = os.open(self.portstr, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK)
PermissionError: [Errno 13] Permission denied: '/dev/ttyACM0'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/tinyprog", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.5/dist-packages/tinyprog/__main__.py", line 298, in main
    with serial.Serial(active_port, timeout=1.0, writeTimeout=1.0) as ser:
  File "/usr/local/lib/python3.5/dist-packages/serial/serialutil.py", line 240, in __init__
    self.open()
  File "/usr/local/lib/python3.5/dist-packages/serial/serialposix.py", line 268, in open
    raise SerialException(msg.errno, "could not open port {}: {}".format(self._port, msg))
serial.serialutil.SerialException: [Errno 13] could not open port /dev/ttyACM0: [Errno 13] Permission denied: '/dev/ttyACM0'

Going to have to make sure I am in the group for that I suppose.

[gudenau]

    TinyProg CLI
    ------------
    Using device id 1d50:6130
    Only one board with active bootloader, using it.
    Programming /dev/ttyACM0 with hardware.bin
Traceback (most recent call last):
  File "/usr/local/bin/tinyprog", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.5/dist-packages/tinyprog/__main__.py", line 309, in main
    addr = fpga.meta.userimage_addr_range()[0]
  File "/usr/local/lib/python3.5/dist-packages/tinyprog/__init__.py", line 95, in userimage_addr_range
    return self._get_addr_range(u"userimage")
  File "/usr/local/lib/python3.5/dist-packages/tinyprog/__init__.py", line 101, in _get_addr_range
    addr_str = self.root[u"bootmeta"][u"addrmap"][name]
TypeError: 'NoneType' object is not subscriptable

And now it is this.

[tinyfpga]

Ok, thanks! I’m going to see if I can reproduce this.

On Sun, Jul 15, 2018 at 1:11 PM gudenau notifications@github.com wrote:

TinyProg CLI
------------
Using device id 1d50:6130
Only one board with active bootloader, using it.
Programming /dev/ttyACM0 with hardware.bin

Traceback (most recent call last): File "/usr/local/bin/tinyprog", line 11, in sys.exit(main()) File "/usr/local/lib/python3.5/dist-packages/tinyprog/main.py", line 309, in main addr = fpga.meta.userimage_addr_range()[0] File "/usr/local/lib/python3.5/dist-packages/tinyprog/init.py", line 95, in userimage_addr_range return self._get_addr_range(u"userimage") File "/usr/local/lib/python3.5/dist-packages/tinyprog/init.py", line 101, in _get_addr_range addr_str = self.root[u"bootmeta"][u"addrmap"][name] TypeError: 'NoneType' object is not subscriptable

And now it is this.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tinyfpga/TinyFPGA-BX/issues/2#issuecomment-405114960, or mute the thread https://github.com/notifications/unsubscribe-auth/AcnVg4HwZ9AWwpJPHaR5ZkJjxL-bWiUtks5uG6HTgaJpZM4VP-_N .

[tinyfpga]

Can you run tinyprog -m and paste the result?

[gudenau]

Traceback (most recent call last):
  File "/usr/local/bin/tinyprog", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.5/dist-packages/tinyprog/__main__.py", line 229, in main
    m["port"] = port
TypeError: 'NoneType' object does not support item assignment

[gudenau]

I wonder if it is because Python 3 is called python3 and not python.

[tinyfpga]

Ok, I wrote a script to help debug what's going on. It will dump out the raw metadata from your board and attempt to parse it in json. Any Python stack traces will be dumped out as well.

First, you'll need to install the hexdump python module:

pip install hexdump

Then you'll need to save the raw_meta_dump.py file I've attached and run it with python:

raw_meta_dump.zip

python3 raw_meta_dump.py

Please send me all the output once you run this.

[tinyfpga]

You might need to use sudo pip3 install hexdump

[gudenau]

['/dev/ttyACM0']
Raw metadata:

1
00000000: 7B 22 62 6F 61 72 64 6D  65 74 61 22 3A 7B 22 6E  {"boardmeta":{"n
00000010: 61 6D 65 22 3A 22 54 69  6E 79 46 50 47 41 20 42  ame":"TinyFPGA B
00000020: 58 22 2C 22 66 70 67 61  22 3A 22 69 63 65 34 30  X","fpga":"ice40
00000030: 6C 70 38 6B 2D 63 6D 38  31 22 2C 22 68 76 65 72  lp8k-cm81","hver
00000040: 22 3A 22 31 2E 30 2E 30  22 2C 22 75 75 69 64 22  ":"1.0.0","uuid"
00000050: 3A 22 XX XX XX XX XX XX  XX XX XX XX XX XX XX XX  :"XXXXXXXX-XXXX-
00000060: XX XX XX XX XX XX XX XX  XX XX XX XX XX XX XX XX  XXXX-XXXX-XXXXXX
00000070: XX XX XX XX XX XX 22 7D  7D 00 20 20 20 20 20 20  XXXXXX"}}.      
00000080: 20 20 20 20 20 20 20 20  20 20 20 20 20 20 20 20                  
00000090: 20 20 20 20 20 20 20 20  20 20 20 20 20 20 20 20                  
000000A0: 20 20 20 20 20 20 20 20  20 20 20 20 20 20 20 20                  
000000B0: 20 20 20 20 20 20 20 20  20 20 20 20 20 20 20 20                  
000000C0: 20 20 20 20 20 20 20 20  20 20 20 20 20 20 20 20                  
000000D0: 20 20 20 20 20 20 20 20  20 20 20 20 20 20 20 20                  
000000E0: 20 20 20 20 20 20 20 20  20 20 20 20 20 20 20 20                  
000000F0: 20 20 20 20 20 20 20 20  20 20 20 20 20 20 20                    

2
00000000: 7B 22 62 6F 6F 74 6D 65  74 61 22 3A 7B 22 62 6F  {"bootmeta":{"bo
00000010: 6F 74 6C 6F 61 64 65 72  22 3A 22 54 69 6E 79 46  otloader":"TinyF
00000020: 50 47 41 20 55 53 42 20  42 6F 6F 74 6C 6F 61 64  PGA USB Bootload
00000030: 65 72 22 2C 22 62 76 65  72 22 3A 22 31 2E 30 2E  er","bver":"1.0.
00000040: 31 22 2C 22 75 70 64 61  74 65 22 3A 22 68 74 74  1","update":"htt
00000050: 70 73 3A 2F 2F 74 69 6E  79 66 70 67 61 2E 63 6F  ps://tinyfpga.co
00000060: 6D 2F 75 70 64 61 74 65  2F 74 69 6E 79 66 70 67  m/update/tinyfpg
00000070: 61 2D 62 78 22 2C 22 61  64 64 72 6D 61 70 22 3A  a-bx","addrmap":
00000080: 7B 22 62 6F 6F 74 6C 6F  61 64 65 72 22 3A 22 30  {"bootloader":"0
00000090: 78 30 30 30 61 30 2D 30  78 32 38 30 30 30 22 2C  x000a0-0x28000",
000000A0: 22 75 73 65 72 69 6D 61  67 65 22 3A 22 30 78 32  "userimage":"0x2
000000B0: 38 30 30 30 2D 30 78 35  30 30 30 30 22 2C 22 75  8000-0x50000","u
000000C0: 73 65 72 64 61 74 61 22  3A 22 30 78 35 30 30 30  serdata":"0x5000
000000D0: 30 2D 30 78 31 30 30 30  30 30 22 7D 7D 7D 00 20  0-0x100000"}}}. 
000000E0: 20 20 20 20 20 20 20 20  20 20 20 20 20 20 20 20                  
000000F0: 20 20 20 20 20 20 20 20  20 20 20 20 20 20 20                    

3
00000000: FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF  ................
00000010: FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF  ................
00000020: FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF  ................
00000030: FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF  ................
00000040: FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF  ................
00000050: FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF  ................
00000060: FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF  ................
00000070: FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF  ................
00000080: FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF  ................
00000090: FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF  ................
000000A0: FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF  ................
000000B0: FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF  ................
000000C0: FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF  ................
000000D0: FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF  ................
000000E0: FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF  ................
000000F0: FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF     ...............
Attempting to parse to JSON:

1
Traceback (most recent call last):
  File "raw_meta_dump.py", line 42, in <module>
    meta = json.loads(security_register.replace(b"\x00", b"").replace(b"\xff", b""))
  File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
    s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'

The JSON looks fairly okay to me.

[tinyfpga]

That is so weird. I'm testing with the exact same version of Python and I don't get that error. I'm assuming you Xed out the UUID, correct?

[tinyfpga]

Ok, I'm going to try converting the data to unicode. Try the updated script I'm attaching and send back the output. If this updated script works on your machine as it does on mine, then I'll be making a tinyprog update with a fix shortly.

raw_meta_dump_unicode.zip

[gudenau]

I did X that out, since it is a UUID.

--snip--
Attempting to parse to JSON:

1
{
  "boardmeta": {
    "uuid": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
    "hver": "1.0.0",
    "name": "TinyFPGA BX",
    "fpga": "ice40lp8k-cm81"
  }
}

2
{
  "bootmeta": {
    "update": "https://tinyfpga.com/update/tinyfpga-bx",
    "addrmap": {
      "userimage": "0x28000-0x50000",
      "bootloader": "0x000a0-0x28000",
      "userdata": "0x50000-0x100000"
    },
    "bootloader": "TinyFPGA USB Bootloader",
    "bver": "1.0.1"
  }
}

3
Traceback (most recent call last):
  File "raw_meta_dump_unicode.py", line 42, in <module>
    meta = json.loads(bytes(security_register.replace(b"\x00", b"").replace(b"\xff", b"")).decode("utf-8"))
  File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Edit: What platform are you using to test?

[tinyfpga]

Great. Try out the latest tinyprog release:

pip install --upgrade --no-cache-dir "tinyprog>=1.0.11"

[gudenau]

No change with that last script.

[tinyfpga]

Ok, I’ll have to try and reproduce this on real hardware. What’s the exact Linux distribution you are using?

[gudenau]

Debian x64 4.9, I really do think this might be because of Python 2 being called python and Python 3 being called python3.

[tinyfpga]

It's extremely unlikely that the python executable name is affecting the outcome. Can you copy the exact output of the following commands?

pip install --upgrade --no-cache-dir "tinyprog>=1.0.11"
tinyprog -l

[gudenau]

gudenau@DESKTOP-DEBIAN:~$ pip install --upgrade --no-cache-dir "tinyprog>=1.0.11"
bash: pip: command not found
gudenau@DESKTOP-DEBIAN:~$ tinyprog -l

    TinyProg CLI
    ------------
    Using device id 1d50:6130
    Only one board with active bootloader, using it.
    Boards with active bootloaders:

        /dev/ttyACM0: TinyFPGA BX 1.0.0
            UUID: 57c8ba7b-6f2b-4d7b-879b-4127a8c95fcd
            FPGA: ice40lp8k-cm81
gudenau@DESKTOP-DEBIAN:~$ 

[tinyfpga]

That's actually great. It fixed the original problem of parsing the JSON.

Let's try splitting apart the tasks so we can identify where the issue is if there still is one. Go to the project directory and build it with APIO:

apio build

Now try to update the hardware.bin file created by APIO directly using tinyprog:

tinyprog -p hardware.bin

That should work. If it does work, then you can try uploading with APIO:

apio upload

[gudenau]

gudenau@DESKTOP-DEBIAN:~/fpga/blink_project$ apio build
[Mon Jul 16 13:16:13 2018] Processing TinyFPGA-BX
--------------------------------------------------------------------------------
========================= [SUCCESS] Took 0.38 seconds =========================
gudenau@DESKTOP-DEBIAN:~/fpga/blink_project$ tinyprog -p hardware.bin 

    TinyProg CLI
    ------------
    Using device id 1d50:6130
    Only one board with active bootloader, using it.
    Programming /dev/ttyACM0 with hardware.bin
    Programming at addr 028000
    Waking up SPI flash
    135100 bytes to program
    Erasing: 100%|████████████████████████████| 135k/135k [00:01<00:00, 111kB/s]
    Writing: 100%|████████████████████████████| 135k/135k [00:00<00:00, 167kB/s]
    Reading: 100%|████████████████████████████| 135k/135k [00:00<00:00, 493kB/s]
    Success!
gudenau@DESKTOP-DEBIAN:~/fpga/blink_project$ apio upload
Error: board TinyFPGA-BX not connected
gudenau@DESKTOP-DEBIAN:~/fpga/blink_project$ apio upload
Error: the JSON object must be str, not 'bytes'
gudenau@DESKTOP-DEBIAN:~/fpga/blink_project$

With colors of course, but I don't think markup supports that.

Edit: At least with this knowledge I can work on stuff, just annoying.

Edit 2: It is blinking SOS by the way.

[tinyfpga]

Ok, so you are able to program the TinyFPGA board successfully then.

After you program the board, the bootloader is no longer active, your user program is running. In order to program the board again you have to press the reset button.

Now the fact that APIO update still reports an error is interesting. Could be an APIO bug and unrelated to tinyprog.

In the meantime, you can use APIO to build the bitstream, and tinyprog to program it. I think you should file an issue on APIO and reference this one: https://github.com/FPGAwars/apio/

[gudenau]

It does look like it is calling Python 3 based on the syscalls it is making. Guess I should go find their issue tracker and link here?

[tinyfpga]

Yes, I think that's the best course of action: https://github.com/FPGAwars/apio/issues

[gudenau]

FPGAwars/apio#159

[ewenmcneill]

FTR, per https://github.com/FPGAwars/apio/issues/159#issuecomment-406840154 I think the issue is that there's code in only the apio upload functionality that is not Python 3 compatible; the same thing works fine with Python 2. (It seems to me most/all of the rest of apio is already Python 3 compatible, so this looks like an oversight.) So hopefully fixed fairly soon. Meanwhile "use Python 2.7" is probably a useful work around for affected users.

Ewen

[ewenmcneill]

On further investigation (https://github.com/FPGAwars/apio/issues/159#issuecomment-406841647) it appears only Python 3.0 - Python 3.5 will be affected; Python 2.7 defaults to str/bytes conversion, and Python 3.6+ contains a compatibility fix in json.loads() so it will accept bytes.

Ewen