search

feelfreelinux / cspot

A Spotify Connect player targeting, but not limited to embedded devices (ESP32).
Other
490 stars 45 forks source link

Make for ESP32 fails on nanopb build #105

Open BartRijvers11 opened 2 years ago

BartRijvers11 commented 2 years ago

I came accross your github and became enthousiastic ! I want a dedicated embedded spotify player for my home office I have a fresh install of the IDF toolchain on my Windows 10 machine and I updated the GIT repo including submodules. I do get stuck building the nanopb library though. I tried many things for the last couple of days, unfortunately unsuccessful but maybe it's a simple solution I have overlooked. I have ordered a board from AliExpress which is in transit right now and I like to be able to compile the software before I actually receive the board.

Loggings:


C:\Users\bartr\Documents\Arduino\cspot>git submodule update --init --recursive

C:\Users\bartr\Documents\Arduino\cspot>cd targets/esp32

C:\Users\bartr\Documents\Arduino\cspot\targets\esp32>idf.py build
Executing action: all (aliases: build)
Running cmake in directory c:\users\bartr\documents\arduino\cspot\targets\esp32\build
Executing "cmake -G Ninja -DPYTHON_DEPS_CHECKED=1 -DESP_PLATFORM=1 -DIDF_TARGET=esp32 -DCCACHE_ENABLE=1 c:\users\bartr\documents\arduino\cspot\targets\esp32"...
-- Found Git: C:/Users/bartr/esp32/.espressif/tools/idf-git/2.30.1/cmd/git.exe (found version "2.30.1.windows.1")
-- ccache will be used for faster recompilation
-- The C compiler identification is GNU 8.4.0
-- The CXX compiler identification is GNU 8.4.0
-- The ASM compiler identification is GNU
-- Found assembler: C:/Users/bartr/esp32/.espressif/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc.exe
-- Check for working C compiler: C:/Users/bartr/esp32/.espressif/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc.exe
-- Check for working C compiler: C:/Users/bartr/esp32/.espressif/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc.exe -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: C:/Users/bartr/esp32/.espressif/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-g++.exe
-- Check for working CXX compiler: C:/Users/bartr/esp32/.espressif/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-g++.exe -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Project version: e9f1d4e-dirty
-- Building ESP-IDF components for target esp32
-- Project sdkconfig file C:/Users/bartr/Documents/Arduino/cspot/targets/esp32/sdkconfig
Loading defaults file C:/Users/bartr/Documents/Arduino/cspot/targets/esp32/sdkconfig.defaults...
-- Found PythonInterp: C:/Users/bartr/esp32/.espressif/python_env/idf4.1_py3.8_env/Scripts/python.exe (found version "3.8.7")
-- Could NOT find Perl (missing: PERL_EXECUTABLE)
-- Adding linker script C:/Users/bartr/Documents/Arduino/cspot/targets/esp32/build/esp-idf/esp32/esp32_out.ld
-- Adding linker script C:/Users/bartr/esp32/esp-idf/components/esp32/ld/esp32.extram.bss.ld
-- Adding linker script C:/Users/bartr/esp32/esp-idf/components/esp32/ld/esp32.project.ld.in
-- Adding linker script C:/Users/bartr/esp32/esp-idf/components/esp32/ld/esp32.peripherals.ld
-- Adding linker script C:/Users/bartr/esp32/esp-idf/components/esp_rom/esp32/ld/esp32.rom.ld
-- Adding linker script C:/Users/bartr/esp32/esp-idf/components/esp_rom/esp32/ld/esp32.rom.libgcc.ld
-- Adding linker script C:/Users/bartr/esp32/esp-idf/components/esp_rom/esp32/ld/esp32.rom.newlib-data.ld
-- Adding linker script C:/Users/bartr/esp32/esp-idf/components/esp_rom/esp32/ld/esp32.rom.syscalls.ld
-- Found Nanopb: C:/Users/bartr/Documents/Arduino/cspot/cspot/bell/nanopb
CMake Error at C:/Users/bartr/Documents/Arduino/cspot/cspot/bell/nanopb/extra/FindNanopb.cmake:263 (string):
  string sub-command REGEX, mode MATCH needs at least 5 arguments total to
  command.
Call Stack (most recent call first):
  C:/Users/bartr/Documents/Arduino/cspot/cspot/CMakeLists.txt:31 (nanopb_generate_cpp)

CMake Error at C:/Users/bartr/Documents/Arduino/cspot/cspot/bell/nanopb/extra/FindNanopb.cmake:269 (message):
  Your path includes a ':' character used as an option separator for nanopb.
  Upgrade to protoc version >= 3.6.0 or use a different path.
Call Stack (most recent call first):
  C:/Users/bartr/Documents/Arduino/cspot/cspot/CMakeLists.txt:31 (nanopb_generate_cpp)

-- Configuring incomplete, errors occurred!
See also "C:/Users/bartr/Documents/Arduino/cspot/targets/esp32/build/CMakeFiles/CMakeOutput.log".
cmake failed with exit code 1

C:\Users\bartr\Documents\Arduino\cspot\targets\esp32>cmake --version
cmake version 3.13.4

CMake suite maintained and supported by Kitware (kitware.com/cmake).

C:\Users\bartr\Documents\Arduino\cspot\targets\esp32>idf.py --version
ESP-IDF v4.1.2

C:\Users\bartr>python -V
Python 3.9.10```
BartRijvers11 commented 2 years ago

I found a regex bug in the FindNanopb.cmake line 263: the last variable needs quotes to continue building (to the next error). I installed Ubuntu in Virtualbox in the meantime.

HeikoGr commented 2 years ago

you don't need to install a virtualbox machine. WSL2 could help you too.

https://github.com/HeikoGr/euphonium/wiki/using-Windows-Subsystem-for-Linux-(WSL)-for-compilation

this should work for cspot alone as well...

BUT: you can't flash via WSL2 without porblems. so maybe virtualbox is still a good alternative...

BartRijvers11 commented 2 years ago

Thanks ! Since I have worked with ubuntu in the past, it would be one less problem to worry about. I installed the last LTS version just tonight:

bart@bart-VirtualBox:~/cspot/targets/esp32$ uname -a
Linux bart-VirtualBox 5.13.0-27-generic #29~20.04.1-Ubuntu SMP Fri Jan 14 00:32:30 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
bart@bart-VirtualBox:~/cspot/targets/esp32$ protoc --version
libprotoc 3.6.1
bart@bart-VirtualBox:~/cspot/targets/esp32$ idf.py --version
ESP-IDF v5.0-dev-1340-g0027483e93
bart@bart-VirtualBox:~/cspot/targets/esp32$ python --version
Python 3.8.10

Patched locally:

bart@bart-VirtualBox:~/cspot/cspot/bell/nanopb$ git diff
diff --git a/extra/FindNanopb.cmake b/extra/FindNanopb.cmake
index 206f5e6..144326e 100644
--- a/extra/FindNanopb.cmake
+++ b/extra/FindNanopb.cmake
@@ -260,7 +260,7 @@ function(NANOPB_GENERATE_CPP SRCS HDRS)
     # - Or a newer one, using --nanopb_opt which requires a version of protoc >= 3.6
     # So we will determine which version of protoc we have available and choose accordingly.
     execute_process(COMMAND ${PROTOBUF_PROTOC_EXECUTABLE} --version OUTPUT_VARIABLE PROTOC_VERSION_STRING OUTPUT_STRIP_TRAILING_WHITESPACE)
-    string(REGEX MATCH "[(0-9)].*.[(0-9)].*.[(0-9)].*" PROTOC_VERSION ${PROTOC_VERSION_STRING})
+    string(REGEX MATCH "[(0-9)].*.[(0-9)].*.[(0-9)].*" PROTOC_VERSION "${PROTOC_VERSION_STRING}")

     if(PROTOC_VERSION VERSION_LESS "3.6.0")
         #try to use the older way

Now I get as far as building the bootloader but get stuck at the protoc version:

[93/95] Linking C executable bootloader.elf
[94/95] Generating binary image from built executable
esptool.py v3.3-dev
Creating esp32 image...
Merged 1 ELF section
Successfully created esp32 image.
Generated /home/bart/cspot/targets/esp32/build/bootloader/bootloader.bin
[95/95] cd /home/bart/cspot/targets/esp32/build/bootloader/esp-idf/esptool_py && /home/bart/.espressif/python_env/idf5.0_py3.8_env/bin/python /home/bart/esp/esp-idf/components/partition_table/check_sizes.py --offset 0x8000 bootloader 0x1000 /home/bart/cspot/targets/esp32/build/bootloader/bootloader.bin
Bootloader binary size 0x6a40 bytes. 0x5c0 bytes (5%) free.
[591/1292] Running C++ protocol buffer compiler using nanopb plugin on /home/bart/cspot/cspot/protobuf/authentication.proto
FAILED: esp-idf/main/cspot/protobuf/authentication.pb.c esp-idf/main/cspot/protobuf/authentication.pb.h 
cd /home/bart/cspot/targets/esp32/build/esp-idf/main/cspot && /usr/bin/protoc -I/home/bart/cspot/targets/esp32/build/esp-idf/main/cspot/nanopb/generator -I/home/bart/cspot/targets/esp32/build/esp-idf/main/cspot/nanopb/generator/proto -I/home/bart/cspot/targets/esp32/build/esp-idf/main/cspot -I/home/bart/cspot/cspot --plugin=protoc-gen-nanopb=/home/bart/cspot/targets/esp32/build/esp-idf/main/cspot/nanopb/generator/protoc-gen-nanopb "--nanopb_opt= -I/home/bart/cspot/cspot/protobuf -I/home/bart/cspot/cspot" --nanopb_out=/home/bart/cspot/targets/esp32/build/esp-idf/main/cspot /home/bart/cspot/cspot/protobuf/authentication.proto
[libprotobuf WARNING google/protobuf/compiler/parser.cc:562] No syntax specified for the proto file: protobuf/authentication.proto. Please use 'syntax = "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2 syntax.)
Traceback (most recent call last):
  File "/home/bart/cspot/targets/esp32/build/esp-idf/main/cspot/nanopb/generator/nanopb_generator.py", line 51, in <module>
    from .proto import nanopb_pb2
ImportError: attempted relative import with no known parent package

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/bart/cspot/targets/esp32/build/esp-idf/main/cspot/nanopb/generator/nanopb_generator.py", line 72, in <module>
    import proto.nanopb_pb2 as nanopb_pb2
  File "/home/bart/cspot/targets/esp32/build/esp-idf/main/cspot/nanopb/generator/proto/nanopb_pb2.py", line 10, in <module>
    from google.protobuf import symbol_database as _symbol_database
ImportError: cannot import name 'symbol_database' from 'google.protobuf' (/home/bart/.espressif/python_env/idf5.0_py3.8_env/lib/python3.8/site-packages/google/protobuf/__init__.py)
--nanopb_out: protoc-gen-nanopb: Plugin failed with status code 1.
[592/1292] Linking C static library esp-idf/esp_http_server/libesp_http_server.a
ninja: build stopped: subcommand failed.
ninja failed with exit code 1

I wonder why it's so hard to just get started... but as long as I feel that Iḿ making progress I'm willing to play pitpull, bite until I figure it out. Google only provides answers of similar issues reported in 2016 and should be solved by now... Stuck again but that's it for me tonight. Hope someone has some ideas to get me going ?

HeikoGr commented 2 years ago

Did you install the dependencies?

sudo pip3 install protobuf grpcio-tools

maybe you also need to install protobuf-compiler. I can’t test it now.

sudo apt install protobuf-compiler

BartRijvers11 commented 2 years ago

Thanks, Yes, I installed all dependencies mentioned in the main readme.md file: Requirement already satisfied: protobuf in /usr/lib/python3/dist-packages (3.6.1) Requirement already satisfied: grpcio-tools in /usr/local/lib/python3.8/dist-packages (1.43.0) Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from grpcio-tools) (45.2.0) Requirement already satisfied: grpcio>=1.43.0 in /usr/local/lib/python3.8/dist-packages (from grpcio-tools) (1.43.0) Requirement already satisfied: six>=1.5.2 in /usr/lib/python3/dist-packages (from grpcio>=1.43.0->grpcio-tools) (1.14.0)

protobuf-compiler is already the newest version (3.6.1.3-2ubuntu5).

If I put the following line at the top of authentication.proto file:

syntax = "proto2";

(I still don't know why it doesn't work out of the box, am I the only one having these kind of issues ?) the build gives me another error:

[591/1292] Running C++ protocol buffer compiler using nanopb plugin on /home/bart/cspot/cspot/protobuf/authentication.proto
FAILED: esp-idf/main/cspot/protobuf/authentication.pb.c esp-idf/main/cspot/protobuf/authentication.pb.h 
cd /home/bart/cspot/targets/esp32/build/esp-idf/main/cspot && /usr/bin/protoc -I/home/bart/cspot/targets/esp32/build/esp-idf/main/cspot/nanopb/generator -I/home/bart/cspot/targets/esp32/build/esp-idf/main/cspot/nanopb/generator/proto -I/home/bart/cspot/targets/esp32/build/esp-idf/main/cspot -I/home/bart/cspot/cspot --plugin=protoc-gen-nanopb=/home/bart/cspot/targets/esp32/build/esp-idf/main/cspot/nanopb/generator/protoc-gen-nanopb "--nanopb_opt= -I/home/bart/cspot/cspot/protobuf -I/home/bart/cspot/cspot" --nanopb_out=/home/bart/cspot/targets/esp32/build/esp-idf/main/cspot /home/bart/cspot/cspot/protobuf/authentication.proto
Traceback (most recent call last):
  File "/home/bart/cspot/targets/esp32/build/esp-idf/main/cspot/nanopb/generator/nanopb_generator.py", line 51, in <module>
    from .proto import nanopb_pb2
ImportError: attempted relative import with no known parent package

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/bart/cspot/targets/esp32/build/esp-idf/main/cspot/nanopb/generator/nanopb_generator.py", line 72, in <module>
    import proto.nanopb_pb2 as nanopb_pb2
  File "/home/bart/cspot/targets/esp32/build/esp-idf/main/cspot/nanopb/generator/proto/nanopb_pb2.py", line 10, in <module>
    from google.protobuf import symbol_database as _symbol_database
ImportError: cannot import name 'symbol_database' from 'google.protobuf' (/home/bart/.espressif/python_env/idf5.0_py3.8_env/lib/python3.8/site-packages/google/protobuf/__init__.py)
--nanopb_out: protoc-gen-nanopb: Plugin failed with status code 1.
[592/1292] Linking C static library esp-idf/esp_http_server/libesp_http_server.a
ninja: build stopped: subcommand failed.
ninja failed with exit code 1
HeikoGr commented 2 years ago

what version of esp-idf do you use? the (working) github action https://github.com/feelfreelinux/cspot/blob/master/.github/workflows/esp-idf.yml uses version 4.4

anabolyc commented 2 years ago

Please make sure you've installed all the dependencies in the esp-idf virtual environment, not globally. Nornally it is located in the ~/.espressif/python_env/idf4.4_py_someting folder

BartRijvers11 commented 2 years ago

Hello people, I am up to date with https://github.com/espressif/esp-idf.git and have remove the build directory before building. I have run the export shell command in de esp32 directory before installing stuff in the same terminal, I'm not sure how I can install the dependencies in the folder @anabolyc mentions or how I can check that, maybe this is sufficient info:

~/.espressif/python_env/idf5.0_py3.8_env/lib/python3.8/site-packages$ ls
alldefconfig.py                               contextlib2                            idna                         pygdbmi                                 semantic_version-2.8.5.dist-info
allmodconfig.py                               contextlib2-21.6.0.dist-info           idna-3.3.dist-info           pygdbmi-0.9.0.2.dist-info               serial
allnoconfig.py                                cryptography                           kconfiglib-13.7.1.dist-info  pyparsing                               setconfig.py
allyesconfig.py                               cryptography-36.0.1.dist-info          kconfiglib.py                pyparsing-3.0.7.dist-info               setuptools
bitstring-3.1.9.dist-info                     defconfig.py                           libfuturize                  pyserial-3.5.dist-info                  setuptools-60.5.0.dist-info
bitstring.py                                  _distutils_hack                        libpasteurize                python3_protobuf-2.5.0.dist-info        six-1.16.0.dist-info
certifi                                       distutils-precedence.pth               listnewconfig.py             python3_protobuf-2.5.0-py3.8-nspkg.pth  six.py
certifi-2021.10.8.dist-info                   ecdsa                                  menuconfig.py                PyYAML-6.0.dist-info                    tqdm
cffi                                          ecdsa-0.17.0.dist-info                 oldconfig.py                 reedsolo-1.5.4.dist-info                tqdm-4.62.3.dist-info
cffi-1.15.0.dist-info                         elftools                               olddefconfig.py              reedsolo.py                             urllib3
_cffi_backend.cpython-38-x86_64-linux-gnu.so  future                                 past                         requests                                urllib3-1.26.8.dist-info
cffi.libs                                     future-0.18.2.dist-info                pip                          requests-2.27.1.dist-info               _virtualenv.pth
charset_normalizer                            genconfig.py                           pip-21.3.1.dist-info         requests_toolbelt                       _virtualenv.py
charset_normalizer-2.0.10.dist-info           google                                 pkg_resources                requests_toolbelt-0.9.1.dist-info       wheel
click                                         guiconfig.py                           __pycache__                  savedefconfig.py                        wheel-0.37.1.dist-info
click-8.0.3.dist-info                         idf_component_manager                  pycparser                    schema-0.7.5.dist-info                  _yaml
construct                                     idf_component_manager-1.0.1.dist-info  pycparser-2.21.dist-info     schema.py                               yaml
construct-2.10.54.dist-info                   idf_component_tools                    pyelftools-0.27.dist-info    semantic_version
HeikoGr commented 2 years ago

maybe this causes the problems. can you try esp-idf in version 4.4? https://github.com/espressif/esp-idf/tree/v4.4

anabolyc commented 2 years ago

Failed for me as well with IDF5, had to roll back to IDF4.4

anabolyc commented 2 years ago

To be sure that you've installed deps in the venv run commands from venv folder similar to this:

~/.espressif/python_env/idf4.4_py3.8_env$ ./bin/python --version
Python 3.8.10

and so on

HeikoGr commented 2 years ago

Failed for me as well with IDF5, had to roll back to IDF4.4

i just tested idf5 on my cspot clone/fork with github action. the occuring errors don't relate to the problem(s) of @BartRijvers11

working idf4.4: https://github.com/HeikoGr/cspot/actions/runs/1750212376 failing idf5: https://github.com/HeikoGr/cspot/actions/runs/1750222894

HeikoGr commented 2 years ago

@BartRijvers11 : maybe you need to undo your local patch

HeikoGr commented 2 years ago

to be prepared when your order of the esp32 arrived, you could

  1. tell me, which chip you have ordered (to select the correct dac), and i'll set up my fork to copmpile it for you, or
  2. clone the repository yourself, activate github action (one extra click), modify the src, and let github compile cspot for you

how you receive the compiled artifacts: https://github.com/HeikoGr/cspot/blob/HeikoGr-artifacts/.github/workflows/esp-idf.yml

(i will make a PR for this repo)

BartRijvers11 commented 2 years ago

To be sure that you've installed deps in the venv run commands from venv folder similar to this:

~/.espressif/python_env/idf4.4_py3.8_env$ ./bin/python --version
Python 3.8.10

and so on

I've rolled back to IDF 4.4 release branch, reverted local changes and I have the same version of python in that folder. If the venv file is important, can you share what it looks like on your platform and how I can fill missing entries (just add them textually or add using a command) ?

BartRijvers11 commented 2 years ago

Thanks a lot guys, success at last ! I first resorted to building the cli version. I get warnings about the proto version not being set, but it assumes version 2 and completes the build. I then realized with @anabolyc 's remarks that I need to install the python extensions in the IDF toolchain python, not the global Ubuntu python. This did the trick for me to be able to build the ESP software, maybe some clarify can be provided in the readme.md file for newbees like me.

BTW. I ordered an AliExpress board with ES8388 as the AC101 is currently unavailable.

BartRijvers11 commented 2 years ago

(Unrelated to this issue) Could you provide a template authBlob.json and a small description of how I can get that file on my ESP32 ?

HeikoGr commented 2 years ago

as far as i understand authBlob.json stores the login credentials for the cli version. you don't need the file for esp32.

BartRijvers11 commented 2 years ago

Ok, then how can I set op my Spotify credentials in the ESP32 ?

HeikoGr commented 2 years ago

You don’t need to.

You need to compile cspot with your wifi credentials (this is the point, i remember why a precompiled firmware makes no sense, as long as there is no wifi manager included). The Spotify credentials come from the device you connect to the cspot.

HeikoGr commented 2 years ago

@BartRijvers11 Did you succeed?

BartRijvers11 commented 2 years ago

Yes, I got the board 2 days ago and installed a pre-built version of euphonium (0.0.14). It works with webradio and spotify but the volume is very soft and I can't control it. My board has the ES8388 chip and Filip mentioned a new release coming soon. I haven't checked that as I did not yet have the time. Cheers !