Bug in uart binary client

[mrquincle]

Describe the bug

On running make uart_binary_client the build system defaults to errors.

UART client, runs in background
** UART client, keep running in separate shell
Listening for logs on port /dev/ttyACM0, and using files in "/home/anne/workspace/bluenet/source" to find the log formats.
Traceback (most recent call last):
  File "/home/anne/workspace/bluenet/scripts/log-client.py", line 44, in <module>
    bluenetLogs.setSourceFilesDir(sourceFilesDir)
  File "/home/anne/.local/lib/python3.8/site-packages/bluenet_logs/BluenetLogs.py", line 20, in setSourceFilesDir
    self.logLineRetriever.setSourceFilesDir(dir)
  File "/home/anne/.local/lib/python3.8/site-packages/bluenet_logs/LogLineRetriever.py", line 39, in setSourceFilesDir
    self._cacheFileContents()
  File "/home/anne/.local/lib/python3.8/site-packages/bluenet_logs/LogLineRetriever.py", line 82, in _cacheFileContents
    lines = file.readlines()
  File "/usr/lib/python3.8/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd3 in position 7734: invalid continuation byte
make[3]: *** [CMakeFiles/uart_binary_client.dir/build.make:59: CMakeFiles/uart_binary_client] Error 123
make[2]: *** [CMakeFiles/Makefile2:272: CMakeFiles/uart_binary_client.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:279: CMakeFiles/uart_binary_client.dir/rule] Error 2
make: *** [Makefile:229: uart_binary_client] Error 2

To Reproduce

Checkout and run make uart_binary_client.

Expected behavior

Working binary UART client.

Already done

At least on an error like this the user should be hinted in setting the -DDOWNLOAD_BLUENET_LIB_LOGS=ON option for cmake.

Requirement already satisfied: bluenet-logs in /home/anne/.local/lib/python3.8/site-packages (0.1.3)
Requirement already satisfied: crownstone-uart~=2.0 in /home/anne/.local/lib/python3.8/site-packages (from bluenet-logs) (2.0.0)
Requirement already satisfied: crownstone-core>=2.0.0 in /home/anne/.local/lib/python3.8/site-packages (from crownstone-uart~=2.0->bluenet-logs) (2.1.0)
Requirement already satisfied: pyserial>=3.4.0 in /home/anne/.local/lib/python3.8/site-packages (from crownstone-uart~=2.0->bluenet-logs) (3.5)
Requirement already satisfied: pyaes~=1.6 in /home/anne/.local/lib/python3.8/site-packages (from crownstone-core>=2.0.0->crownstone-uart~=2.0->bluenet-logs) (1.6.1)

Not that that fixed it. :-)

[mrquincle]

It looks like every single file in the working directory is opened.

    def _cacheFileContents(self):
        """
        Cache contents of all cached fileNames.
        """
        for fileName in self.fileNames:
            filePath = fileName
            try:
                file = open(filePath, "r", encoding="utf-8")
                lines = file.readlines()
                file.close()
            except:
                _LOGGER.warning(f"Error in file {filePath}")
            self.bluenetFiles[fileName] = lines

Only source files should be inspected.

It currently fails on a tags file which is generated by ctags. However, that's quite irrelevant. There can be other files here as well that are not source files.

[mrquincle]

Options to fix:

Walk the source and include directories

Walk it manually and pick up .c, .cpp and .h files and a few other extensions that might be useful.

Cache files in cmake

Make use of the cache files that are populated by cmake.

cat CMakeFiles/crownstone.dir/CXX.includecache | sort | uniq | grep -v '#'

This lists:

access_config.h
access.h
access_internal.h
access_publish.h
access_reliable.h
access_status.h
...
...
...
util/cs_Utils.h
util/cs_UuidParser.h
util/cs_Variance.h
util/cs_WireFormat.h
utility
utils.h
vector

Here you can use a regexp if you'd like to excluded Nordic files or other files.

Variables in cmake

Alternatively, you can use the CMake variable we have defined to collect all source files:

cat list_variables.txt | grep FOLDER_SOURCE

This does not include header files, so it's less useful.

Tradeoff

Pros and cons.

• Walking the source files means that it doesn't need to compile properly and that you can still checkout a bunch of source and header files and get binary logging output. This removes the dependency on (proper) compilation (of e.g. firmware of a previous release or another branch).
• Walking the source means that are independent on cmake, on cmake internals, on cmake caching, and on (outdated) cmake builds.
• Using the cmake cache means that you can very easily included the GENERATED_SOURCES as well (they are in the build directory of the given target, e.g. build/default/include/cfg/cs_StaticConfig.h).
• Using the cmake cache means you won't accidentally miss a file (or extension, say .hpp files).

Seems to me the first argument outweighs almost everything else.

[mrquincle]

@vliedel This is still not fixed. If there is any binary (or even non-binary) file in the directories, the binary logger can choke on it.