Crash when requesting platforms list over gRPC

arduino / arduino-cli

Arduino command line tool

https://arduino.github.io/arduino-cli/latest/

GNU General Public License v3.0

4.4k stars 384 forks source link

Crash when requesting platforms list over gRPC #1434

Closed 4ntoine closed 3 years ago

4ntoine commented 3 years ago

Bug Report

Current behavior

via gRPC - requesting the list of platforms

it crashes with the stacktrace:

> arduino-cli daemon
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0x47ca4ba]

goroutine 8 [running]:
github.com/arduino/arduino-cli/commands.PlatformReleaseToRPC(0x0, 0x0)
    github.com/arduino/arduino-cli/commands/core.go:31 +0x3a
github.com/arduino/arduino-cli/commands/core.GetPlatforms(0xc00124ab00, 0xc0011e5b00, 0x0, 0xc000559b10, 0x400dd25, 0x4b20a40)
    github.com/arduino/arduino-cli/commands/core/list.go:54 +0x1c5
github.com/arduino/arduino-cli/commands/daemon.(*ArduinoCoreServerImpl).PlatformList(0xc0001b3fc0, 0x4ca1a70, 0xc0012ae3c0, 0xc00124ab00, 0xc0001b3fc0, 0xc0012ae3c0, 0xc000065b80)
    github.com/arduino/arduino-cli/commands/daemon/daemon.go:272 +0x2f
github.com/arduino/arduino-cli/rpc/cc/arduino/cli/commands/v1._ArduinoCoreService_PlatformList_Handler(0x4b413a0, 0xc0001b3fc0, 0x4ca1a70, 0xc0012ae3c0, 0xc001277500, 0x0, 0x4ca1a70, 0xc0012ae3c0, 0xc0010dff90, 0x6)
    github.com/arduino/arduino-cli/rpc/cc/arduino/cli/commands/v1/commands.pb.go:3834 +0x217
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000083380, 0x4cab8d8, 0xc000001b00, 0xc0011e5b00, 0xc00023edb0, 0x5220ea0, 0x0, 0x0, 0x0)
    google.golang.org/grpc@v1.27.0/server.go:1024 +0x522
google.golang.org/grpc.(*Server).handleStream(0xc000083380, 0x4cab8d8, 0xc000001b00, 0xc0011e5b00, 0x0)
    google.golang.org/grpc@v1.27.0/server.go:1313 +0xd4c
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00032e030, 0xc000083380, 0x4cab8d8, 0xc000001b00, 0xc0011e5b00)
    google.golang.org/grpc@v1.27.0/server.go:722 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
    google.golang.org/grpc@v1.27.0/server.go:720 +0xa5

Expected behavior

Expected to have a response

Environment

CLI version (output of arduino-cli version):

> arduino-cli version
arduino-cli alpha Version: 0.18.3 Commit: d710b642 Date: 2021-05-21T17:04:33Z

OS and platform: macOS mojave 10.14.6

Additional context

4ntoine commented 3 years ago

However the following cli request succeeds:

> arduino-cli core list
ID              Installed Latest Name                  
arduino:avr     1.8.3     1.8.3  Arduino AVR Boards    
esp32:esp32     1.0.4     1.0.4  ESP32 Arduino         
esp8266:esp8266 2.7.1     2.7.1  ESP8266 Boards (2.7.1)

per1234 commented 3 years ago

arduino-cli alpha Version: 0.18.3

Hi @4ntoine. Thanks so much for your report. I see you're using a significantly outdated version of Arduino CLI. Please try again with the latest nightly build and then post an update with your results: https://arduino.github.io/arduino-cli/dev/installation/#nightly-builds

Note also that there have been some recent breaking changes to the gRPC API: https://arduino.github.io/arduino-cli/dev/UPGRADING/

4ntoine commented 3 years ago

@per1234 This is the most recent version that is available via brew on mac (just tried). I will try more recent one.

4ntoine commented 3 years ago

Same for 0.19.0:

/tmp/bin asmirnov
> ./arduino-cli version
arduino-cli alpha Version: 0.19.0 Commit: 56419ecd Date: 2021-09-02T14:47:35Z
/tmp/bin asmirnov
> ./arduino-cli daemon
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x47f9d24]

goroutine 36 [running]:
github.com/arduino/arduino-cli/commands.Init(0xc0000af7a0, 0xc0004d3c00, 0x40109f8, 0x10)
    /home/build/commands/instances.go:170 +0x64
github.com/arduino/arduino-cli/commands/daemon.(*ArduinoCoreServerImpl).Init(0xc00038a720, 0xc0000af7a0, 0x4d1b748, 0xc000069750, 0xc00038a720, 0x18)
    /home/build/commands/daemon/daemon.go:226 +0x6a
github.com/arduino/arduino-cli/rpc/cc/arduino/cli/commands/v1._ArduinoCoreService_Init_Handler(0x4b974e0, 0xc00038a720, 0x4d19468, 0xc0007f6240, 0x5310e38, 0xc00024c900)
    /home/build/rpc/cc/arduino/cli/commands/v1/commands_grpc.pb.go:1164 +0x113
google.golang.org/grpc.(*Server).processStreamingRPC(0xc00018ca80, 0x4d1f658, 0xc00049ca80, 0xc00024c900, 0xc000383170, 0x52d21a0, 0x0, 0x0, 0x0)
    /go/pkg/mod/google.golang.org/grpc@v1.37.0/server.go:1464 +0xcd8
google.golang.org/grpc.(*Server).handleStream(0xc00018ca80, 0x4d1f658, 0xc00049ca80, 0xc00024c900, 0x0)
    /go/pkg/mod/google.golang.org/grpc@v1.37.0/server.go:1544 +0xca5
google.golang.org/grpc.(*Server).serveStreams.func1.(0xc0000b4070, 0xc00018ca80, 0x4d1f658, 0xc00049ca80, 0xc00024c900)
    /go/pkg/mod/google.golang.org/grpc@v1.37.0/server.go:878 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
    /go/pkg/mod/google.golang.org/grpc@v1.37.0/server.go:876 +0x1fd
/tmp/bin asmirnov

4ntoine commented 3 years ago

Same for nightly build:

/tmp asmirnov
> curl -fsSL https://raw.githubusercontent.com/arduino/arduino-cli/master/install.sh | sh -s nightly-latest
Installing in /tmp/bin
ARCH=64bit
OS=macOS
Using curl as download tool
Downloading https://downloads.arduino.cc/arduino-cli/nightly/arduino-cli_nightly-latest_macOS_64bit.tar.gz
An existing arduino-cli was found at /usr/local/bin/arduino-cli. Please prepend "/tmp/bin" to your $PATH or remove the existing one.
arduino-cli alpha Version: nightly-20210908 Commit: 7415e26 Date: 2021-09-08T01:25:40Z installed successfully in /tmp/bin
/tmp asmirnov
> cd bin/
/tmp/bin asmirnov
> ./arduino-cli version
arduino-cli alpha Version: nightly-20210908 Commit: 7415e26 Date: 2021-09-08T01:25:40Z
/tmp/bin asmirnov
> ./arduino-cli daemon
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x4815d44]

goroutine 10 [running]:
github.com/arduino/arduino-cli/commands.Init(0xc00044c8d0, 0xc000147c00, 0x4010738, 0x10)
    /home/build/commands/instances.go:171 +0x64
github.com/arduino/arduino-cli/commands/daemon.(*ArduinoCoreServerImpl).Init(0xc0002d5d90, 0xc00044c8d0, 0x4d43cc8, 0xc000466bb0, 0xc0002d5d90, 0x18)
    /home/build/commands/daemon/daemon.go:226 +0x6a
github.com/arduino/arduino-cli/rpc/cc/arduino/cli/commands/v1._ArduinoCoreService_Init_Handler(0x4bb6f00, 0xc0002d5d90, 0x4d418c8, 0xc0000123c0, 0x5346b18, 0xc000305680)
    /home/build/rpc/cc/arduino/cli/commands/v1/commands_grpc.pb.go:1164 +0x113
google.golang.org/grpc.(*Server).processStreamingRPC(0xc000258a80, 0x4d47c38, 0xc000082a80, 0xc000305680, 0xc00044c270, 0x5308100, 0x0, 0x0, 0x0)
    /go/pkg/mod/google.golang.org/grpc@v1.38.0/server.go:1533 +0xcd8
google.golang.org/grpc.(*Server).handleStream(0xc000258a80, 0x4d47c38, 0xc000082a80, 0xc000305680, 0x0)
    /go/pkg/mod/google.golang.org/grpc@v1.38.0/server.go:1613 +0xca5
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc00012cd50, 0xc000258a80, 0x4d47c38, 0xc000082a80, 0xc000305680)
    /go/pkg/mod/google.golang.org/grpc@v1.38.0/server.go:934 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
    /go/pkg/mod/google.golang.org/grpc@v1.38.0/server.go:932 +0x1fd

4ntoine commented 3 years ago

I've noted now (for 0.19.0 and later) it fails during the init() (that's probably due to breaking changes you've mentioned). I will update my .proto file and retry

4ntoine commented 3 years ago

Yup, i've regenerated the source code (for 0.19.0) and i can see version request succeeds, but init fails. Here are the tests (Dart):

import 'package:arduino_cli_dart/cc/arduino/cli/commands/v1/commands.pbgrpc.dart';
import 'package:flutter_test/flutter_test.dart';

import 'package:grpc/grpc.dart';

void main() {
  ClientChannel _getChannel() => ClientChannel(
    'localhost',
    port: 50051,
    options: const ChannelOptions(credentials: ChannelCredentials.insecure()),
  );

  ArduinoCoreServiceClient _getClient() => ArduinoCoreServiceClient(_getChannel());

  test(' version request',  () async {
    final channel = _getChannel();
    final client = _getClient();
    var versionResponse = await client.version(VersionRequest());
    final version = versionResponse.version;
    print('Arduino cli version is: $version');
    await channel.shutdown();
  });

  test('init request',  () async {
    final channel = _getChannel();
    final client = ArduinoCoreServiceClient(channel);
    var initResponse = await client.init(InitRequest.getDefault()).single;
    expect(initResponse.hasError(), false);
    await channel.shutdown();
  });
}

I can provide access to the repo if necessary.

4ntoine commented 3 years ago

jic (trying "0.19.0" with test):

/tmp asmirnov
> curl -fsSL https://raw.githubusercontent.com/arduino/arduino-cli/master/install.sh | sh -s 0.19.0
Installing in /tmp/bin
ARCH=64bit
OS=macOS
Using curl as download tool
Downloading https://downloads.arduino.cc/arduino-cli/arduino-cli_0.19.0_macOS_64bit.tar.gz
An existing arduino-cli was found at /usr/local/bin/arduino-cli. Please prepend "/tmp/bin" to your $PATH or remove the existing one.
arduino-cli alpha Version: 0.19.0 Commit: 56419ecd Date: 2021-09-02T14:47:35Z installed successfully in /tmp/bin
/tmp asmirnov
> cd bin/
/tmp/bin asmirnov
> ./arduino-cli daemon
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x47f9d24]

goroutine 66 [running]:
github.com/arduino/arduino-cli/commands.Init(0xc0004961b0, 0xc000061c00, 0x40109f8, 0x10)
    /home/build/commands/instances.go:170 +0x64
github.com/arduino/arduino-cli/commands/daemon.(*ArduinoCoreServerImpl).Init(0xc000454740, 0xc0004961b0, 0x4d1b748, 0xc0000a4080, 0xc000454740, 0x18)
    /home/build/commands/daemon/daemon.go:226 +0x6a
github.com/arduino/arduino-cli/rpc/cc/arduino/cli/commands/v1._ArduinoCoreService_Init_Handler(0x4b974e0, 0xc000454740, 0x4d19468, 0xc0004ae000, 0x5310e38, 0xc0004a4000)
    /home/build/rpc/cc/arduino/cli/commands/v1/commands_grpc.pb.go:1164 +0x113
google.golang.org/grpc.(*Server).processStreamingRPC(0xc00025aa80, 0x4d1f658, 0xc000471b00, 0xc0004a4000, 0xc00044b1a0, 0x52d21a0, 0x0, 0x0, 0x0)
    /go/pkg/mod/google.golang.org/grpc@v1.37.0/server.go:1464 +0xcd8
google.golang.org/grpc.(*Server).handleStream(0xc00025aa80, 0x4d1f658, 0xc000471b00, 0xc0004a4000, 0x0)
    /go/pkg/mod/google.golang.org/grpc@v1.37.0/server.go:1544 +0xca5
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc00003afd0, 0xc00025aa80, 0x4d1f658, 0xc000471b00, 0xc0004a4000)
    /go/pkg/mod/google.golang.org/grpc@v1.37.0/server.go:878 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
    /go/pkg/mod/google.golang.org/grpc@v1.37.0/server.go:876 +0x1fd
/tmp/bin asmirnov

4ntoine commented 3 years ago

Here is the repo to reproduce. Here is the test. The proto files are in the dir. To run the test flutter test.

Let me know if i need to update anything or missing anything (or just doing the wrong way).

silvanocerza commented 3 years ago

Interesting, I'll investigate. Thanks for all the feedback.

Cool to see a project in Flutter use the Arduino CLI. :)

umbynos commented 3 years ago

I opened another issue to track the homebrew problem https://github.com/arduino/arduino-cli/issues/1435

silvanocerza commented 3 years ago

@4ntoine the gRPC interface changed quite a bit with 0.19.0, the initialization step has been split in two parts, first you must Create an Instance and only then you can Init it.

It should be something like this in Dart.

    var createResponse = await client.create(CreateRequest.getDefault());
    var initRequest = InitRequest.create();
    initRequest.instance = createResponse.instance;
    var initResponse = await client.init(initRequest).single;

This change has been documented in the Upgrading guide: https://arduino.github.io/arduino-cli/0.19/UPGRADING/#change-of-behaviour-of-grpc-init-function

If you manage to fix this let us know if there are still issues with the PlatformList function.

4ntoine commented 3 years ago

@silvanocerza Thanks for the explanation! I will try it

4ntoine commented 3 years ago

It works now, thanks! PS. However i'd suggest to be more careful with breaking backward compatibility

4ntoine commented 3 years ago

Ahh, i was too happy when init() started working. Now GetPlatform() fails (with "0.19.0"): The test:

test('get platforms',  () async {
    final channel = _getChannel();
    final client = ArduinoCoreServiceClient(channel);
    var createResponse = await client.create(CreateRequest.getDefault());
    var initResponse = await client.init(InitRequest(
        instance: createResponse.instance
    )).single;

    final platformsResponse = await client.platformList(PlatformListRequest(
      instance: createResponse.instance,
      all: true
    ));

    await channel.shutdown();
  });

The crash:

/tmp/bin asmirnov
> ./arduino-cli core list
Error initializing instance: loading platform release esp8266:esp8266@2.7.1: loading boards: skipping loading of boards esp8266:esp8266:espduino: malformed custom board options
ID              Installed Latest Name                  
arduino:avr     1.8.3     1.8.3  Arduino AVR Boards    
esp32:esp32     1.0.4     1.0.4  ESP32 Arduino         
esp8266:esp8266 2.7.1     2.7.1  ESP8266 Boards (2.7.1)

/tmp/bin asmirnov
> ./arduino-cli daemon
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x78 pc=0x47f34ba]

goroutine 20 [running]:
github.com/arduino/arduino-cli/commands.PlatformReleaseToRPC(0x0, 0x0)
    /home/build/commands/core.go:31 +0x3a
github.com/arduino/arduino-cli/commands/core.GetPlatforms(0xc0048fe040, 0xc000639b00, 0x0, 0xc0001a7b30, 0x400e125, 0x4b74380)
    /home/build/commands/core/list.go:48 +0x1ec
github.com/arduino/arduino-cli/commands/daemon.(*ArduinoCoreServerImpl).PlatformList(0xc000446600, 0x4d14a70, 0xc001d04030, 0xc0048fe040, 0xc000446600, 0xc001d04030, 0xc00024fba0)
    /home/build/commands/daemon/daemon.go:314 +0x2f
github.com/arduino/arduino-cli/rpc/cc/arduino/cli/commands/v1._ArduinoCoreService_PlatformList_Handler(0x4b974e0, 0xc000446600, 0x4d14a70, 0xc001d04030, 0xc001d08000, 0x0, 0x4d14a70, 0xc001d04030, 0xc001d02008, 0x6)
    /home/build/rpc/cc/arduino/cli/commands/v1/commands_grpc.pb.go:1683 +0x217
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000210a80, 0x4d1f658, 0xc000001380, 0xc000639b00, 0xc00043eed0, 0x52d04c0, 0x0, 0x0, 0x0)
    /go/pkg/mod/google.golang.org/grpc@v1.37.0/server.go:1217 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc000210a80, 0x4d1f658, 0xc000001380, 0xc000639b00, 0x0)
    /go/pkg/mod/google.golang.org/grpc@v1.37.0/server.go:1540 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc00018a2b0, 0xc000210a80, 0x4d1f658, 0xc000001380, 0xc000639b00)
    /go/pkg/mod/google.golang.org/grpc@v1.37.0/server.go:878 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
    /go/pkg/mod/google.golang.org/grpc@v1.37.0/server.go:876 +0x1fd
/tmp/bin asmirnov
> ./arduino-cli version
arduino-cli alpha Version: 0.19.0 Commit: 56419ecd Date: 2021-09-02T14:47:35Z

4ntoine commented 3 years ago

yup, same with nightly:

> curl -fsSL https://raw.githubusercontent.com/arduino/arduino-cli/master/install.sh | sh -s nightly-latest
Installing in /tmp/bin
ARCH=64bit
OS=macOS
Using curl as download tool
Downloading https://downloads.arduino.cc/arduino-cli/nightly/arduino-cli_nightly-latest_macOS_64bit.tar.gz
An existing arduino-cli was found at /usr/local/bin/arduino-cli. Please prepend "/tmp/bin" to your $PATH or remove the existing one.
arduino-cli alpha Version: nightly-20210908 Commit: 7415e26 Date: 2021-09-08T01:25:40Z installed successfully in /tmp/bin
/tmp asmirnov
> 
/tmp asmirnov
> cd bin/
/tmp/bin asmirnov
> ./arduino-cli version
arduino-cli alpha Version: nightly-20210908 Commit: 7415e26 Date: 2021-09-08T01:25:40Z
/tmp/bin asmirnov
> ./arduino-cli daemon
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x78 pc=0x480f4fa]

goroutine 68 [running]:
github.com/arduino/arduino-cli/commands.PlatformReleaseToRPC(0x0, 0x0)
    /home/build/commands/core.go:31 +0x3a
github.com/arduino/arduino-cli/commands/core.GetPlatforms(0xc0031e4040, 0xc0001bea20, 0x0, 0xc00013bb30, 0x400de65, 0x4b93c60)
    /home/build/commands/core/list.go:48 +0x1ec
github.com/arduino/arduino-cli/commands/daemon.(*ArduinoCoreServerImpl).PlatformList(0xc000285d20, 0x4d3cc90, 0xc0031ea030, 0xc0031e4040, 0xc000285d20, 0xc0031ea030, 0xc000848ba0)
    /home/build/commands/daemon/daemon.go:314 +0x2f
github.com/arduino/arduino-cli/rpc/cc/arduino/cli/commands/v1._ArduinoCoreService_PlatformList_Handler(0x4bb6f00, 0xc000285d20, 0x4d3cc90, 0xc0031ea030, 0xc0031ee000, 0x0, 0x4d3cc90, 0xc0031ea030, 0xc0031e8008, 0x6)
    /home/build/rpc/cc/arduino/cli/commands/v1/commands_grpc.pb.go:1683 +0x217
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000208a80, 0x4d47c38, 0xc000001800, 0xc0001bea20, 0xc0003f81e0, 0x5306400, 0x0, 0x0, 0x0)
    /go/pkg/mod/google.golang.org/grpc@v1.38.0/server.go:1286 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc000208a80, 0x4d47c38, 0xc000001800, 0xc0001bea20, 0x0)
    /go/pkg/mod/google.golang.org/grpc@v1.38.0/server.go:1609 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc000124190, 0xc000208a80, 0x4d47c38, 0xc000001800, 0xc0001bea20)
    /go/pkg/mod/google.golang.org/grpc@v1.38.0/server.go:934 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
    /go/pkg/mod/google.golang.org/grpc@v1.38.0/server.go:932 +0x1fd

4ntoine commented 3 years ago

Note i haven't fully migrated (but i guess the message is enough (same that i get with cli) and it should not crash anyway).

silvanocerza commented 3 years ago

I had to change your tests a bit since I had some issues with async, this is what I ran:

  test('init request', () async {
    final channel = _getChannel();
    final client = ArduinoCoreServiceClient(channel);
    var createResponse = await client.create(CreateRequest.getDefault());
    await for (var res
        in client.init(InitRequest(instance: createResponse.instance))) {
      expect(res.hasError(), false);
    }
    await channel.shutdown();
  });

  test('get platforms', () async {
    final channel = _getChannel();
    final client = ArduinoCoreServiceClient(channel);
    var createResponse = await client.create(CreateRequest.getDefault());
    await for (var res
        in client.init(InitRequest(instance: createResponse.instance))) {
      expect(res.hasError(), false);
    }

    final platformsResponse = await client.platformList(
        PlatformListRequest(instance: createResponse.instance, all: true));

    for (var platform in platformsResponse.installedPlatforms) {
      print(platform.id);
    }

    await channel.shutdown();
  });

And this is the output:

$ flutter test
00:01 +0:  version request                                                                                                                        
Arduino cli version is: 0.19.0
00:02 +2: get platforms                                                                                                                           
adafruit:avr
adafruit:nrf52
adafruit:samd
TeeOnArdu:avr
adafruit:wiced
arduino:avr
arduino:mbed_edge
arduino:mbed_nano
arduino:mbed_portenta
arduino:mbed_rp2040
arduino:megaavr
arduino:nrf52
arduino:sam
arduino:samd
Arrow:samd
atmel-avr-xminis:avr
emoro:avr
esp32:esp32
industruino:samd
Intel:arc32
Intel:i586
Intel:i686
arcore:avr
littleBits:avr
SparkFun:apollo3
SparkFun:avr
SparkFun:esp32
esp8266:esp8266
SparkFun:samd
SPRESENSE:spresense
Microsoft:win10
arduino:mbed
00:02 +3: All tests passed!

I can't seem to reproduce the issue, the request returns correctly.

Could you show the output of arduino-cli config dump? Maybe there's some package index that has issues?

4ntoine commented 3 years ago

@silvanocerza There is obviously some difference in set up (probably platforms).

> ./arduino-cli config dump
board_manager:
  additional_urls: []
daemon:
  port: "50051"
directories:
  data: /Users/asmirnov/Library/Arduino15
  downloads: /Users/asmirnov/Library/Arduino15/staging
  user: /Users/asmirnov/Documents/Arduino
library:
  enable_unsafe_install: false
logging:
  file: ""
  format: text
  level: info
metrics:
  addr: :9090
  enabled: true
output:
  no_color: false
sketch:
  always_export_binaries: false
updater:
  enable_notification: true

Anything just looking into the source code at the lines where is crashes?

silvanocerza commented 3 years ago

I see your board_manager.additional_urls are empty:

board_manager:
  additional_urls: []

How did you install esp32:esp32 and esp8266:esp8266? Did you remove the URLs after installing them?

Also can you run daemon with the --verbose flag? It gives much more information.

Anything just looking into the source code at the lines where is crashes?

Simply put the CLI can't find the latest release of a platform, no idea which nor why it can't find it. I can technically already fix it but I first want to know the root cause for a more thourough fix.

4ntoine commented 3 years ago

@silvanocerza

Nope, i did not remove anything intentionally, i can still see it in IDE settings:

> ./arduino-cli daemon --verbose
INFO[0000] Config file not found, using default values  
INFO[0000] arduino-cli version nightly-20210908         
INFO[0000] Setting up Prometheus metrics on :9090/metrics 
INFO[0000] Starting daemon on TCP address 127.0.0.1:50051 
INFO[0000] Daemon is now listening on 127.0.0.1:50051... 
INFO[0020] Checking if CLI is Bundled into the IDE      
INFO[0020] Adding libraries dir                          dir=/Users/asmirnov/Documents/Arduino/libraries location=user
INFO[0020] Checking signature                            index=/Users/asmirnov/Library/Arduino15/package_index.json signatureFile=/Users/asmirnov/Library/Arduino15/package_index.json.sig trusted=true
INFO[0020] Loading hardware from: /Users/asmirnov/Library/Arduino15/packages 
INFO[0020] Loading package arduino from: /Users/asmirnov/Library/Arduino15/packages/arduino/hardware 
INFO[0020] Checking signature                            error="opening signature file: open /Users/asmirnov/Library/Arduino15/packages/arduino/hardware/avr/1.8.3/installed.json.sig: no such file or directory" index=/Users/asmirnov/Library/Arduino15/packages/arduino/hardware/avr/1.8.3/installed.json signatureFile=/Users/asmirnov/Library/Arduino15/packages/arduino/hardware/avr/1.8.3/installed.json.sig
INFO[0020] Loaded platform                               platform="arduino:avr@1.8.3"
INFO[0020] Checking existence of 'tools' path: /Users/asmirnov/Library/Arduino15/packages/arduino/tools 
INFO[0020] Loading tools from dir: /Users/asmirnov/Library/Arduino15/packages/arduino/tools 
INFO[0020] Loaded tool                                   tool="arduino:arduinoOTA@1.3.0"
INFO[0020] Loaded tool                                   tool="arduino:avr-gcc@4.8.1-arduino5"
INFO[0020] Loaded tool                                   tool="arduino:avr-gcc@7.3.0-atmel3.6.1-arduino7"
INFO[0020] Loaded tool                                   tool="arduino:avrdude@6.3.0-arduino17"
INFO[0020] Loading package builtin from: /Users/asmirnov/Library/Arduino15/packages/builtin 
INFO[0020] Checking existence of 'tools' path: /Users/asmirnov/Library/Arduino15/packages/builtin/tools 
INFO[0020] Loading tools from dir: /Users/asmirnov/Library/Arduino15/packages/builtin/tools 
INFO[0020] Loaded tool                                   tool="builtin:ctags@5.8-arduino11"
INFO[0020] Loaded tool                                   tool="builtin:mdns-discovery@0.9.2"
INFO[0020] Loaded tool                                   tool="builtin:serial-discovery@1.0.0"
INFO[0020] Loaded tool                                   tool="builtin:serial-discovery@1.2.1"
INFO[0020] Loaded tool                                   tool="builtin:serial-discovery@1.3.0-rc1"
INFO[0020] Loading package digistump from: /Users/asmirnov/Library/Arduino15/packages/digistump/hardware 
INFO[0020] Checking existence of 'tools' path: /Users/asmirnov/Library/Arduino15/packages/digistump/tools 
INFO[0020] Loading tools from dir: /Users/asmirnov/Library/Arduino15/packages/digistump/tools 
INFO[0020] Loaded tool                                   tool="digistump:micronucleus@2.0a4"
INFO[0020] Loading package esp32 from: /Users/asmirnov/Library/Arduino15/packages/esp32/hardware 
INFO[0020] Loaded platform                               platform="esp32:esp32@1.0.3"
INFO[0020] Loaded platform                               platform="esp32:esp32@1.0.4"
INFO[0020] Checking existence of 'tools' path: /Users/asmirnov/Library/Arduino15/packages/esp32/tools 
INFO[0020] Loading tools from dir: /Users/asmirnov/Library/Arduino15/packages/esp32/tools 
INFO[0020] Loaded tool                                   tool="esp32:esptool_py@2.6.1"
INFO[0020] Loaded tool                                   tool="esp32:mkspiffs@0.2.3"
INFO[0020] Loaded tool                                   tool="esp32:xtensa-esp32-elf-gcc@1.22.0-80-g6c4433a-5.2.0"
INFO[0020] Loading package esp8266 from: /Users/asmirnov/Library/Arduino15/packages/esp8266/hardware 
INFO[0020] Checking existence of 'tools' path: /Users/asmirnov/Library/Arduino15/packages/esp8266/tools 
INFO[0020] Loading tools from dir: /Users/asmirnov/Library/Arduino15/packages/esp8266/tools 
INFO[0020] Loaded tool                                   tool="esp8266:mklittlefs@2.5.0-4-fe5bb56"
INFO[0020] Loaded tool                                   tool="esp8266:mkspiffs@2.5.0-4-b40a506"
INFO[0020] Loaded tool                                   tool="esp8266:python3@3.7.2-post1"
INFO[0020] Loaded tool                                   tool="esp8266:xtensa-lx106-elf-gcc@2.5.0-4-b40a506"
INFO[0020] Adding libraries dir                          dir=/Users/asmirnov/Library/Arduino15/packages/esp8266/hardware/esp8266/2.7.1/libraries location=platform
INFO[0020] Adding libraries dir                          dir=/Users/asmirnov/Library/Arduino15/packages/arduino/hardware/avr/1.8.3/libraries location=platform
INFO[0020] Adding libraries dir                          dir=/Users/asmirnov/Library/Arduino15/packages/esp32/hardware/esp32/1.0.4/libraries location=platform
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x78 pc=0x480f4fa]

goroutine 24 [running]:
github.com/arduino/arduino-cli/commands.PlatformReleaseToRPC(0x0, 0x0)
    /home/build/commands/core.go:31 +0x3a
github.com/arduino/arduino-cli/commands/core.GetPlatforms(0xc004c1a040, 0xc00016d200, 0x0, 0xc0000c5b30, 0x400de65, 0x4b93c60)
    /home/build/commands/core/list.go:48 +0x1ec
github.com/arduino/arduino-cli/commands/daemon.(*ArduinoCoreServerImpl).PlatformList(0xc00020fce0, 0x4d3cc90, 0xc004c20030, 0xc004c1a040, 0xc00020fce0, 0xc004c20030, 0xc0004bbba0)
    /home/build/commands/daemon/daemon.go:314 +0x2f
github.com/arduino/arduino-cli/rpc/cc/arduino/cli/commands/v1._ArduinoCoreService_PlatformList_Handler(0x4bb6f00, 0xc00020fce0, 0x4d3cc90, 0xc004c20030, 0xc004c24000, 0x0, 0x4d3cc90, 0xc004c20030, 0xc004c1e008, 0x6)
    /home/build/rpc/cc/arduino/cli/commands/v1/commands_grpc.pb.go:1683 +0x217
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00018ea80, 0x4d47c38, 0xc00050a180, 0xc00016d200, 0xc000402270, 0x5306400, 0x0, 0x0, 0x0)
    /go/pkg/mod/google.golang.org/grpc@v1.38.0/server.go:1286 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc00018ea80, 0x4d47c38, 0xc00050a180, 0xc00016d200, 0x0)
    /go/pkg/mod/google.golang.org/grpc@v1.38.0/server.go:1609 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0004c2000, 0xc00018ea80, 0x4d47c38, 0xc00050a180, 0xc00016d200)
    /go/pkg/mod/google.golang.org/grpc@v1.38.0/server.go:934 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
    /go/pkg/mod/google.golang.org/grpc@v1.38.0/server.go:932 +0x1fd

silvanocerza commented 3 years ago

Aha! You installed them with the IDE, this is something that I didn't consider.

4ntoine commented 3 years ago

Yup, but i installed 1.8.2 via IDE and then upgraded it to 1.8.3 via gRPC.

4ntoine commented 3 years ago

BTW i'm having this:

> ./arduino-cli core list
Error initializing instance: loading platform release esp8266:esp8266@2.7.1: loading boards: skipping loading of boards esp8266:esp8266:espduino: malformed custom board options
ID              Installed Latest Name                  
arduino:avr     1.8.3     1.8.3  Arduino AVR Boards    
esp32:esp32     1.0.4     1.0.4  ESP32 Arduino         
esp8266:esp8266 2.7.1     2.7.1  ESP8266 Boards (2.7.1)

Can it be a reason?

4ntoine commented 3 years ago

I've uninstalled all the toolchains using cli and installed "arduino:avr" again. Unfortunately i'm still having the crash. Not sure what i can check:

> ./arduino-cli core list
ID          Installed Latest Name              
arduino:avr 1.8.3     1.8.3  Arduino AVR Boards

/tmp/bin asmirnov
> 
/tmp/bin asmirnov
> ./arduino-cli daemon --verbose
INFO[0000] Config file not found, using default values  
INFO[0000] arduino-cli version 0.19.0                   
INFO[0000] Setting up Prometheus metrics on :9090/metrics 
INFO[0000] Starting daemon on TCP address 127.0.0.1:50051 
INFO[0000] Daemon is now listening on 127.0.0.1:50051... 
INFO[0010] Checking if CLI is Bundled into the IDE      
INFO[0010] Adding libraries dir                          dir=/Users/asmirnov/Documents/Arduino/libraries location=user
INFO[0010] Checking signature                            index=/Users/asmirnov/Library/Arduino15/package_index.json signatureFile=/Users/asmirnov/Library/Arduino15/package_index.json.sig trusted=true
INFO[0010] Loading hardware from: /Users/asmirnov/Library/Arduino15/packages 
INFO[0010] Loading package arduino from: /Users/asmirnov/Library/Arduino15/packages/arduino/hardware 
INFO[0010] Checking signature                            error="opening signature file: open /Users/asmirnov/Library/Arduino15/packages/arduino/hardware/avr/1.8.3/installed.json.sig: no such file or directory" index=/Users/asmirnov/Library/Arduino15/packages/arduino/hardware/avr/1.8.3/installed.json signatureFile=/Users/asmirnov/Library/Arduino15/packages/arduino/hardware/avr/1.8.3/installed.json.sig
INFO[0010] Loaded platform                               platform="arduino:avr@1.8.3"
INFO[0010] Checking existence of 'tools' path: /Users/asmirnov/Library/Arduino15/packages/arduino/tools 
INFO[0010] Loading tools from dir: /Users/asmirnov/Library/Arduino15/packages/arduino/tools 
INFO[0010] Loaded tool                                   tool="arduino:arduinoOTA@1.3.0"
INFO[0010] Loaded tool                                   tool="arduino:avr-gcc@4.8.1-arduino5"
INFO[0010] Loaded tool                                   tool="arduino:avr-gcc@7.3.0-atmel3.6.1-arduino7"
INFO[0010] Loaded tool                                   tool="arduino:avrdude@6.3.0-arduino17"
INFO[0010] Loading package builtin from: /Users/asmirnov/Library/Arduino15/packages/builtin 
INFO[0010] Checking existence of 'tools' path: /Users/asmirnov/Library/Arduino15/packages/builtin/tools 
INFO[0010] Loading tools from dir: /Users/asmirnov/Library/Arduino15/packages/builtin/tools 
INFO[0010] Loaded tool                                   tool="builtin:ctags@5.8-arduino11"
INFO[0010] Loaded tool                                   tool="builtin:mdns-discovery@0.9.2"
INFO[0010] Loaded tool                                   tool="builtin:serial-discovery@1.0.0"
INFO[0010] Loaded tool                                   tool="builtin:serial-discovery@1.2.1"
INFO[0010] Loaded tool                                   tool="builtin:serial-discovery@1.3.0-rc1"
INFO[0010] Loading package digistump from: /Users/asmirnov/Library/Arduino15/packages/digistump/hardware 
INFO[0010] Checking existence of 'tools' path: /Users/asmirnov/Library/Arduino15/packages/digistump/tools 
INFO[0010] Loading tools from dir: /Users/asmirnov/Library/Arduino15/packages/digistump/tools 
INFO[0010] Loaded tool                                   tool="digistump:micronucleus@2.0a4"
INFO[0010] Loading package esp32 from: /Users/asmirnov/Library/Arduino15/packages/esp32/hardware 
INFO[0010] Checking existence of 'tools' path: /Users/asmirnov/Library/Arduino15/packages/esp32/tools 
INFO[0010] Loading tools from dir: /Users/asmirnov/Library/Arduino15/packages/esp32/tools 
INFO[0010] Loaded tool                                   tool="esp32:esptool_py@2.6.1"
INFO[0010] Loaded tool                                   tool="esp32:mkspiffs@0.2.3"
INFO[0010] Loaded tool                                   tool="esp32:xtensa-esp32-elf-gcc@1.22.0-80-g6c4433a-5.2.0"
INFO[0010] Loading package esp8266 from: /Users/asmirnov/Library/Arduino15/packages/esp8266/hardware 
INFO[0010] Checking existence of 'tools' path: /Users/asmirnov/Library/Arduino15/packages/esp8266/tools 
INFO[0010] Loading tools from dir: /Users/asmirnov/Library/Arduino15/packages/esp8266/tools 
INFO[0010] Loaded tool                                   tool="esp8266:mklittlefs@2.5.0-4-fe5bb56"
INFO[0010] Loaded tool                                   tool="esp8266:mkspiffs@2.5.0-4-b40a506"
INFO[0010] Loaded tool                                   tool="esp8266:python3@3.7.2-post1"
INFO[0010] Loaded tool                                   tool="esp8266:xtensa-lx106-elf-gcc@2.5.0-4-b40a506"
INFO[0010] Adding libraries dir                          dir=/Users/asmirnov/Library/Arduino15/packages/arduino/hardware/avr/1.8.3/libraries location=platform
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x78 pc=0x47f34ba]

goroutine 41 [running]:
github.com/arduino/arduino-cli/commands.PlatformReleaseToRPC(0x0, 0x0)
    /home/build/commands/core.go:31 +0x3a
github.com/arduino/arduino-cli/commands/core.GetPlatforms(0xc002dc6000, 0xc000155440, 0x0, 0xc0003b3b30, 0x400e125, 0x4b74380)
    /home/build/commands/core/list.go:48 +0x1ec
github.com/arduino/arduino-cli/commands/daemon.(*ArduinoCoreServerImpl).PlatformList(0xc00035e540, 0x4d14a70, 0xc002dc4000, 0xc002dc6000, 0xc00035e540, 0xc002dc4000, 0xc000068ba0)
    /home/build/commands/daemon/daemon.go:314 +0x2f
github.com/arduino/arduino-cli/rpc/cc/arduino/cli/commands/v1._ArduinoCoreService_PlatformList_Handler(0x4b974e0, 0xc00035e540, 0x4d14a70, 0xc002dc4000, 0xc002dc2000, 0x0, 0x4d14a70, 0xc002dc4000, 0xc002dc0000, 0x6)
    /home/build/rpc/cc/arduino/cli/commands/v1/commands_grpc.pb.go:1683 +0x217
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000018a80, 0x4d1f658, 0xc0000b4180, 0xc000155440, 0xc000352f00, 0x52d04c0, 0x0, 0x0, 0x0)
    /go/pkg/mod/google.golang.org/grpc@v1.37.0/server.go:1217 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc000018a80, 0x4d1f658, 0xc0000b4180, 0xc000155440, 0x0)
    /go/pkg/mod/google.golang.org/grpc@v1.37.0/server.go:1540 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0000d8030, 0xc000018a80, 0x4d1f658, 0xc0000b4180, 0xc000155440)
    /go/pkg/mod/google.golang.org/grpc@v1.37.0/server.go:878 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
    /go/pkg/mod/google.golang.org/grpc@v1.37.0/server.go:876 +0x1fd

4ntoine commented 3 years ago

I can confirm it does not crash with your code, but crashes with initializing only the first Create response. However i believe it should not crash anyway.. Feel free to close if you think it's an expected behaviour

cmaglie commented 3 years ago

From the stacktrace, the null pointer exception is here https://github.com/arduino/arduino-cli/blob/7415e269061ff4d90e6356f6c1156941122d58c3/commands/core.go#L31 it could be that platformRelease.Boards is nil or even platformRelease itself is nil (that would be really strange).

@4ntoine could you send a zip of your ~/.arduino15 and sketchbook folder to examine it? (c.maglie at arduino.cc)

4ntoine commented 3 years ago

@cmaglie here you are https://dropmefiles.com/tm10S available for 14 days (it's 0.5Gb large)

silvanocerza commented 3 years ago

BTW i'm having this:

> ./arduino-cli core list
Error initializing instance: loading platform release esp8266:esp8266@2.7.1: loading boards: skipping loading of boards esp8266:esp8266:espduino: malformed custom board options
ID              Installed Latest Name                  
arduino:avr     1.8.3     1.8.3  Arduino AVR Boards    
esp32:esp32     1.0.4     1.0.4  ESP32 Arduino         
esp8266:esp8266 2.7.1     2.7.1  ESP8266 Boards (2.7.1)

Can it be a reason?

Nah, that's just a warning that certain boards are not loaded because their properties are not correct.

From the stacktrace, the null pointer exception is here

https://github.com/arduino/arduino-cli/blob/7415e269061ff4d90e6356f6c1156941122d58c3/commands/core.go#L31 it could be that platformRelease.Boards is nil or even platformRelease itself is nil (that would be really strange).

len() of a nil slice is 0 so platformRelease must be nil and not platformRelease.Boards, we're in really strange territory. :D

@cmaglie here you are https://dropmefiles.com/tm10S available for 14 days (it's 0.5Gb large)

@4ntoine I've tested with your Arduino15 folder but it keeps working for me, all tests are passing.

I need a reliable way to reproduce the issue before fixing it. :confused:

$ flutter test                   
Building flutter tool...
Downloading package sky_engine...                                  306ms
Downloading flutter_patched_sdk tools...                           378ms
Downloading flutter_patched_sdk_product tools...                   333ms
Downloading linux-x64 tools...                                   2,442ms
Downloading linux-x64/font-subset tools...                         376ms
Running "flutter pub get" in arduino_cli_dart...                 1,624ms
00:10 +0:  version request                                                                      
Arduino cli version is: git-snapshot
00:11 +2: get platforms                                                                         
arduino:avr
arduino:mbed_edge
arduino:mbed_nano
arduino:mbed_portenta
arduino:mbed_rp2040
arduino:megaavr
arduino:nrf52
arduino:sam
arduino:samd
Arrow:samd
atmel-avr-xminis:avr
emoro:avr
industruino:samd
Intel:arc32
Intel:i586
Intel:i686
littleBits:avr
Microsoft:win10
arduino:mbed
00:11 +3: All tests passed!

Could you maybe upload the Arduino folder too? It should contain a libraries folder and other Sketch folders.

4ntoine commented 3 years ago

Unfortunately i've uninstalled "esp32" and "esp8266" toolchains so the state is different to previous.. We can close it and i will reopen if it happens again

silvanocerza commented 3 years ago

Let's leave it open, I want to find the cause of this issue.

Could you try with the attached version of the CLI? I added more logging on certain places, it should give us more information on what's happening or which direction to go to best understand the issue.

arduino-cli_0.19.0_macOS_64bit.tar.gz

per1234 commented 3 years ago

Hi all. I think I might have ran into the same bug.

Necessary conditions

An empty architecture folder under a 3rd party vendor folder is present
- This is a common condition because the architecture folder is not cleaned up by arduino-cli core uninstall
arduino-cli core list uses the --all flag

Steps to reproduce

per@HAL MINGW64 /c/program-files/arduino/cli/arduino-cli_nightly
$ ./arduino-cli core install attiny:avr --additional-urls https://raw.githubusercontent.com/damellis/attiny/ide-1.6.x-boards-manager/package_damellis_attiny_index.json
Downloading packages...
attiny:avr@1.0.2 already downloaded
Installing platform attiny:avr@1.0.2...
Configuring platform....
Platform attiny:avr@1.0.2 installed

per@HAL MINGW64 /c/program-files/arduino/cli/arduino-cli_nightly
$ ./arduino-cli core uninstall attiny:avr
Uninstalling attiny:avr@1.0.2...
Platform attiny:avr@1.0.2 uninstalled

per@HAL MINGW64 /c/program-files/arduino/cli/arduino-cli_nightly
$ ./arduino-cli core list --all
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x90 pc=0xde8109]

goroutine 1 [running]:
github.com/arduino/arduino-cli/commands.PlatformReleaseToRPC(0x0, 0x0)
        /home/build/commands/core.go:31 +0x49
github.com/arduino/arduino-cli/commands/core.GetPlatforms(0xc0000d9d58, 0xc000000004, 0xc0000d9d48, 0x1, 0x1, 0xa)
        /home/build/commands/core/list.go:48 +0x1ec
github.com/arduino/arduino-cli/cli/core.runListCommand(0xc000a56f00, 0xc0008f3430, 0x0, 0x1)
        /home/build/cli/core/list.go:55 +0xfc
github.com/spf13/cobra.(*Command).execute(0xc000a56f00, 0xc0008f3420, 0x1, 0x1, 0xc000a56f00, 0xc0008f3420)
        /go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:860 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0xc0008db680, 0x1, 0x1, 0x0)
        /go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:974 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
        /go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:902
main.main()
        /home/build/main.go:31 +0xea

Additional information

per@HAL MINGW64 /c/program-files/arduino/cli/arduino-cli_nightly
$ ./arduino-cli version
arduino-cli.exe alpha Version: nightly-20211014 Commit: ba8c9b7 Date: 2021-10-14T01:27:50Z

per@HAL MINGW64 /c/program-files/arduino/cli/arduino-cli_nightly
$ tree ~/AppData/Local/Arduino15/packages
C:/Users/per/AppData/Local/Arduino15/packages
|-- attiny
|   `-- hardware
|       `-- avr
`-- builtin
    `-- tools
        |-- ctags
        |   `-- 5.8-arduino11
        |       `-- ctags.exe
        |-- mdns-discovery
        |   `-- 0.9.2
        |       |-- LICENSE.txt
        |       `-- mdns-discovery.exe
        |-- serial-discovery
        |   `-- 1.3.0-rc1
        |       |-- LICENSE.txt
        |       `-- serial-discovery.exe
        `-- serial-monitor
            `-- 0.9.1
                |-- LICENSE.txt
                `-- serial-monitor.exe

13 directories, 7 files

umbynos commented 3 years ago

hi @4ntoine, @per1234 could you please try the build here This problem should be fixed in #1519

per1234 commented 3 years ago

@umbynos it's working correctly now for me when I run the command line process I described in my previous reply. Thanks!