Packaging arduino with upstream avr-gcc or avrdude question

NicoHood commented 7 years ago

Hey guys, I recently got a TU for archlinux which means I can now package arduino for archlinux. The question came up if archlinux should link avr-gcc and/or avrdude and/or arduino-builder with the distribution/upstream packages.

I'd like to present my opinion on this, but also would like to hear other opinions, as I might be wrong. It would be nice if one of you guys could answer, as I think you should know it best: @cmaglie @facchinm @sandeepmistry @matthijskooijman @focalintend @kriegsman

My opinion is to not package the upstream tools.

Compilation size differs
Assemler code differs, makes it hard to debug
Compiler specific code as in fastled could break
New avrlibc definitions might give problems
Reverting of avrdude already showed such problems
Atmel also uses 4.9.x which is quite new if you ask me
If you really want the latest avr-gcc you can use a makefile or change platform.txt

As a side note: We currently package avrdude 6.3 because of a dependency issue, but that is only temporary.

Edit: I've linked all the binaries to the upstream avr-gcc and such errors then occur:

MouseAPI.h:43:19: error: flexible array member in union
  uint16_t whole16[];

This might be a feature of the new avr-gcc but the library itself is broken. And if you use the library manager, its quite trickey to always fix the bugs yourself. And now tell the maintainer to fix those bugs for you avr-gcc 6.x just because you are using arch and the whole world uses the bundled avr-gcc. Those error appear with my own HID library. (I should fix this anyways).

Edit2: And with other sketches I got LTO problems. This might be because I am linking to the wrong pathes or the platform.txt also needs modifications because it uses the wrong parameters.

lto1: internal compiler error: Segmentation fault
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://gcc.gnu.org/bugs.html> for instructions.
lto-wrapper: fatal error: /usr/share/arduino/hardware/tools/avr/bin/avr-gcc returned 1 exit status
compilation terminated.
/usr/bin/avr-ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status

matthijskooijman commented 7 years ago

@NicoHood, your arguments sound sane to me. However, does Arch have any policy on this? I know that Debian has a policy of only allowing a single piece of software to be present in one package, if other software needs it, the other software should use a dependency (I think the rationale is mostly for security, this prevents missing a copy of some vulnerable software hidden inside another package).

plantarum commented 7 years ago

Is it even possible to package Arduino for redistribution? Debian has been stuck on version 1.05 for ages due to unresolved licensing issues: https://github.com/arduino/Arduino/pull/2703

NicoHood commented 7 years ago

So I've still been trying to package arduino with the upstream avr-gcc. I had to solve some issues first, now the packaging can continue.

Its a good practise to download all files via makepkg and not within the build script itself. Its quite a lot to download and then i found the light-bundle option. I've tried it with ant clean build -Dlight_bundle=true, but that gave me some errors when launching the IDE in the 2nd run:

java.io.FileNotFoundException: /usr/share/arduino/hardware/package_index_bundled.json (No such file or directory)
    at java.io.FileInputStream.open0(Native Method)
    at java.io.FileInputStream.open(FileInputStream.java:195)
    at java.io.FileInputStream.<init>(FileInputStream.java:138)
    at cc.arduino.contributions.packages.ContributionsIndexer.parseIndex(ContributionsIndexer.java:194)
    at cc.arduino.contributions.packages.ContributionsIndexer.parseIndex(ContributionsIndexer.java:86)
    at processing.app.BaseNoGui.initPackages(BaseNoGui.java:612)
    at processing.app.Base.<init>(Base.java:292)
    at processing.app.Base.guardedMain(Base.java:219)
    at processing.app.Base.main(Base.java:132)

If I see it correct the light version does not download:

avr-gcc nor avrdude
no builtin libraries
no avr-core

If I get it right this could all be downloadable via an arduino avr-core update, isnt it?

I don't really know for what this is used, but it could be handy for packing arduino for several distributions. The libraries are not essentially required if you ask me. You can redownload those via the boards manager. Most of the time they are out of date anyways. The avrcore can also be redownloaded, as the sam core is also not included by default. I am not sure if this would then also pull in the bundled avr-gcc or not, but it would be cool to use the system avr-gcc if its set by default.

So can anyone help me to explain this feature a bit more?

Edit: I've copied the missing json file and created the /usr/share/arduino/libraries folder which was missing and it now compiles the code as expected. The avr-gcc is the bundled 4.9.x which is downloaded per user. The package is way smaller now and it all works quite good. Maybe those minor bugs could be fixed and this packaging method can be used for all linux distributions from now on?

The fix could look like this:

  <!-- copy hardware folder -->
  <target name="assemble-hardware" depends="assemble-hardware-full, assemble-hardware-light" />
  <target name="assemble-hardware-full" unless="light_bundle">
    <copy todir="${target.path}/hardware">
      <fileset dir="../hardware">
        <exclude name="arduino/sam/**"/>
      </fileset>
    </copy>
  </target>
  <target name="assemble-hardware-light" if="light_bundle">
    <copy file="../hardware/package_index_bundled.json" todir="${target.path}/hardware" />
  </target>

  <!-- copy library folder -->
  <target name="assemble-libraries" depends="assemble-libraries-full, assemble-libraries-light" />
  <target name="assemble-libraries-full" unless="light_bundle">
    <copy todir="${target.path}/libraries">
      <fileset dir="../libraries"/>
    </copy>

    <download-library name="Firmata" version="2.5.3"/>
    <download-library name="Bridge" version="1.6.3"/>
    <download-library name="Robot_Control" version="1.0.2"/>
    <download-library name="Robot_Motor" version="1.0.2"/>
    <download-library name="RobotIRremote" version="1.0.2"/>
    <download-library name="SpacebrewYun" version="1.0.1"/>
    <download-library name="Temboo" version="1.1.7"/>
    <download-library name="Esplora" version="1.0.4"/>
    <download-library name="Mouse" version="1.0.1"/>
    <download-library name="Keyboard" version="1.0.1"/>
  </target>
  <target name="assemble-libraries-light" if="light_bundle">
    <mkdir dir="${target.path}/libraries" />
  </target>

@plantarum this really should be fixed, you are right. Also I've seen other distributions removing some wifi java code, as its not free. I will also look into this.

Salamandar commented 7 years ago

Hi there ! It'd be great to use the upstream avr-gcc, avr-libc, avrdude. That would mean a lighter Arduino package, and, of course, simpler management + understanding of the Arduino toolchain ("uh, why do I need this GCC version ?").

But… Well I'm on Archlinux, and the Arduino package is being splitted. And Marlin (for instance) does not work anymore : It compiles but does not seem to execute.

I downloaded the Arduino IDE, and using the arduino-builder and avrdude from community BUT avr-gcc libc from the Arduino IDE, that works.

/usr/share/arduino/hardware/archlinux-arduino/avr/platform.txt :

- compiler.path=/usr/bin/
+ compiler.path=/home/salamandar/Téléchargements/arduino-1.6.12/hardware/tools/avr/bin/

Maybe avr-gcc6 is incompatible with the arduino hardware libs ? Maybe you should provide a Gcc4.9 package ?

NicoHood commented 7 years ago

@Salamandar It is still possible. See: https://wiki.archlinux.org/index.php/arduino#AVR_Boards

arduino / Arduino

Packaging arduino with upstream avr-gcc or avrdude question #5379