search

platformio / platformio-core

Your Gateway to Embedded Software Development Excellence :alien:
https://platformio.org
Apache License 2.0
7.97k stars 796 forks source link

Implement diff for pio remote compile #825

Closed sticilface closed 7 years ago

sticilface commented 8 years ago

Currently all the working directory files are copied each time it is compiled remotely. It would be fantastic if diff perhaps through rsync could be used to only copy changed files. pio remote could also cache the files for a certain amount of time, to avoid recompilation of unchanged files.

:)

ivankravets commented 8 years ago

PlatformIO Plus 0.6.0 is out! Please pio update on the both sides. Would be thankful for any feedback.

Also, I added new option -d, --working-dir for pio remote agent start. You can set it to /dev/shm and PIO will keep sync things in RAM. In the final results it should work crazy fast :)

sticilface commented 8 years ago

does not seem to accept the -d parameter

Platform Manager
================
Platform espressif8266
--------
Updating espressif8266 @ 1.2.1:     [Up-to-date]
Updating framework-arduinoespressif8266 @ 1.20300.1:    [Up-to-date]
Updating tool-esptool @ 1.409.0:    [Up-to-date]
Updating tool-espotapy @ 1.0.0:     [Up-to-date]
Updating toolchain-xtensa @ 1.40802.0:  [Up-to-date]

Updating tool-pioplus @ 0.6.0:  [Up-to-date]
Updating pysite-pioplus @ 0.3.0:    [Up-to-date]

Library Manager
===============
root@pio:~ # platformio remote agent start -d /dev/shm
Error: no such option: -d
root@pio:~ #
ivankravets commented 8 years ago

Please update http://docs.platformio.org/en/stable/installation.html#development-version

sticilface commented 8 years ago

OK that worked sorry, i thought update worked.

so getting some weird errors... its looking for files in the /src directory when all the files are in the directory with the same name to keep the arduinoide happy.

➜  MyMelvanimate git:(master) ✗ pio remote run -r -e serial
PlatformIO Plus (https://pioplus.com) v0.6.0
[Errno 2] No such file or directory: '/Users/amelvin/git/MyMelvanimate/src/.DS_Store'
➜  MyMelvanimate git:(master) ✗ touch ./src/.DS_Store
➜  MyMelvanimate git:(master) ✗ pio remote run -r -e serial
PlatformIO Plus (https://pioplus.com) v0.6.0
[Errno 2] No such file or directory: '/Users/amelvin/git/MyMelvanimate/src/webeditor.h'
➜  MyMelvanimate git:(master) ✗

you can see its looking for .DS_Store... which if i place in ./src it then looks for the webeditor.h which is in the project root folder, specified in the platformio.ini file. my guess it that it is generating the file list using the platformio.ini data_dir = ./MyMelvanimate/data but then the copy sync bit is defaulting to /src...

ivankravets commented 8 years ago

Thanks for the report. Could you try v0.6.1?

Host: pio update Update agent remotely: pio remote update && pio remote agent reload.

sticilface commented 8 years ago

ok, gave it a bash. working 13hr days at the moment so not much time. It has jumped straight to 0.6.2.

its getting there, however, now it borks as libs included in the libs folder are not being copied. It installs the libs specified in the .ini file. 

➜  MySONOFF git:(master) ✗ pio remote run -r -v -e default
PlatformIO Plus (https://pioplus.com) v0.6.2
[Mon Nov 28 21:56:44 2016] Processing default (build_flags: -Wl,-Tesp8266.flash.4m.ld, lib_deps: ArduinoJson, ESPAsyncTCP, ESPAsyncWebServer, AsyncMqttClient, Bounce2, board_f_cpu: 160000000L, platform: espressif8266, board: esp12e, framework: arduino, board_flash_mode: qio, upload_port: 192.168.1.196, board_f_flash: 80000000L)
--------------------------------------------------------------------------------
Library Storage: /mnt/ramdrive/projects/MySONOFF-3569aba0d50347e8ed8362a338a2905eb8baddf5/.piolibdeps
LibraryManager: Installing id=64
ArduinoJson @ 5.7.2 is already installed
LibraryManager: Installing id=305
ESPAsyncTCP @ 1.0.0 is already installed
LibraryManager: Installing id=306
ESPAsyncWebServer @ 9bdf7c42b4 is already installed
LibraryManager: Installing id=346
AsyncMqttClient @ 0.5.0 is already installed
LibraryManager: Installing id=1106
Bounce2 @ 2.1 is already installed
xtensa-lx106-elf-g++ -o "/mnt/ramdrive/projects/MySONOFF-3569aba0d50347e8ed8362a338a2905eb8baddf5/src/MySONOFF.ino.cpp" -x c++ -fpreprocessed -dD -E "/tmp/tmpePYNRx"
/mnt/ramdrive/projects/MySONOFF-3569aba0d50347e8ed8362a338a2905eb8baddf5/src/MySONOFF.ino:51:0: warning: "_Debugf" redefined [enabled by default]
#define _Debugf(...) {}
^
/mnt/ramdrive/projects/MySONOFF-3569aba0d50347e8ed8362a338a2905eb8baddf5/src/MySONOFF.ino:48:0: note: this is the location of the previous definition
#define _Debugf(...) DEBUG_CODE.printf(__VA_ARGS__)
^
Collected 29 compatible libraries
Looking for dependencies...
Skip platform incompatible dependency {u'platforms': [u'espressif'], u'frameworks': [u'arduino'], u'name': u'ESPAsyncTCP', u'authors': u'Hristo Gochkov'}
Library Dependency Graph
|-- <ArduinoJson> v5.7.2 (/mnt/ramdrive/projects/MySONOFF-3569aba0d50347e8ed8362a338a2905eb8baddf5/.piolibdeps/ArduinoJson_ID64)
|-- <ESPAsyncTCP> v1.0.0 (/mnt/ramdrive/projects/MySONOFF-3569aba0d50347e8ed8362a338a2905eb8baddf5/.piolibdeps/ESPAsyncTCP_ID305)
|   |-- <ESP8266WiFi> v1.0 (/root/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi)
|-- <ESPAsyncWebServer> (/mnt/ramdrive/projects/MySONOFF-3569aba0d50347e8ed8362a338a2905eb8baddf5/.piolibdeps/ESPAsyncWebServer_ID306)
|   |-- <ESPAsyncTCP> v1.0.0 (/mnt/ramdrive/projects/MySONOFF-3569aba0d50347e8ed8362a338a2905eb8baddf5/.piolibdeps/ESPAsyncTCP_ID305)
|   |   |-- <ESP8266WiFi> v1.0 (/root/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi)
|   |-- <ArduinoJson> v5.7.2 (/mnt/ramdrive/projects/MySONOFF-3569aba0d50347e8ed8362a338a2905eb8baddf5/.piolibdeps/ArduinoJson_ID64)
|   |-- <ESP8266WiFi> v1.0 (/root/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi)
|   |-- <Hash> v1.0 (/root/.platformio/packages/framework-arduinoespressif8266/libraries/Hash)
|-- <AsyncMqttClient> v0.5.0 (/mnt/ramdrive/projects/MySONOFF-3569aba0d50347e8ed8362a338a2905eb8baddf5/.piolibdeps/AsyncMqttClient_ID346)
|   |-- <ESPAsyncTCP> v1.0.0 (/mnt/ramdrive/projects/MySONOFF-3569aba0d50347e8ed8362a338a2905eb8baddf5/.piolibdeps/ESPAsyncTCP_ID305)
|   |   |-- <ESP8266WiFi> v1.0 (/root/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi)
|-- <Bounce2> v2.1 (/mnt/ramdrive/projects/MySONOFF-3569aba0d50347e8ed8362a338a2905eb8baddf5/.piolibdeps/Bounce2_ID1106)
|-- <ArduinoOTA> v1.0 (/root/.platformio/packages/framework-arduinoespressif8266/libraries/ArduinoOTA)
|   |-- <ESP8266WiFi> v1.0 (/root/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi)
|   |-- <ESP8266mDNS> (/root/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266mDNS)
|   |   |-- <ESP8266WiFi> v1.0 (/root/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi)
|-- <Hash> v1.0 (/root/.platformio/packages/framework-arduinoespressif8266/libraries/Hash)
|-- <ESP8266mDNS> (/root/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266mDNS)
|   |-- <ESP8266WiFi> v1.0 (/root/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi)
|-- <ESP8266httpUpdate> v1.1 (/root/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266httpUpdate)
|   |-- <ESP8266WiFi> v1.0 (/root/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi)
|   |-- <ESP8266HTTPClient> v1.1 (/root/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266HTTPClient)
|   |   |-- <ESP8266WiFi> v1.0 (/root/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi)
|-- <ESP8266WiFi> v1.0 (/root/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi)
|-- <ESP8266HTTPClient> v1.1 (/root/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266HTTPClient)
|   |-- <ESP8266WiFi> v1.0 (/root/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi)
xtensa-lx106-elf-g++ -o .pioenvs/default/src/MySONOFF.ino.o -c -fno-rtti -fno-exceptions -std=c++11 -Os -mlongcalls -mtext-section-literals -falign-functions=4 -ffunction-sections -fdata-sections -DARDUINO=20300 -DLWIP_OPEN_SRC -DF_CPU=160000000L -D__ets__ -DICACHE_FLASH -DPLATFORMIO=030200 -DESP8266 -DARDUINO_ARCH_ESP8266 -DARDUINO_ESP8266_ESP12 -U__STRICT_ANSI__ -I/root/.platformio/packages/framework-arduinoespressif8266/tools/sdk/include -I/root/.platformio/packages/framework-arduinoespressif8266/tools/sdk/lwip/include -I/root/.platformio/packages/framework-arduinoespressif8266/cores/esp8266 -I/root/.platformio/packages/framework-arduinoespressif8266/variants/nodemcu -I.piolibdeps/ArduinoJson_ID64 -I/root/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src -I.piolibdeps/ESPAsyncTCP_ID305/src -I/root/.platformio/packages/framework-arduinoespressif8266/libraries/Hash/src -I.piolibdeps/ESPAsyncWebServer_ID306/src -I.piolibdeps/AsyncMqttClient_ID346/src -I.piolibdeps/Bounce2_ID1106 -I/root/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266mDNS -I/root/.platformio/packages/framework-arduinoespressif8266/libraries/ArduinoOTA -I/root/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266HTTPClient/src -I/root/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266httpUpdate/src -Isrc src/MySONOFF.ino.cpp
xtensa-lx106-elf-g++ -o .pioenvs/default/FrameworkArduino/StreamString.o -c -fno-rtti -fno-exceptions -std=c++11 -Os -mlongcalls -mtext-section-literals -falign-functions=4 -ffunction-sections -fdata-sections -DARDUINO=20300 -DLWIP_OPEN_SRC -DF_CPU=160000000L -D__ets__ -DICACHE_FLASH -DPLATFORMIO=030200 -DESP8266 -DARDUINO_ARCH_ESP8266 -DARDUINO_ESP8266_ESP12 -U__STRICT_ANSI__ -I/root/.platformio/packages/framework-arduinoespressif8266/tools/sdk/include -I/root/.platformio/packages/framework-arduinoespressif8266/tools/sdk/lwip/include -I/root/.platformio/packages/framework-arduinoespressif8266/cores/esp8266 -I/root/.platformio/packages/framework-arduinoespressif8266/variants/nodemcu /root/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/StreamString.cpp
xtensa-lx106-elf-g++ -o .pioenvs/default/FrameworkArduino/Tone.o -c -fno-rtti -fno-exceptions -std=c++11 -Os -mlongcalls -mtext-section-literals -falign-functions=4 -ffunction-sections -fdata-sections -DARDUINO=20300 -DLWIP_OPEN_SRC -DF_CPU=160000000L -D__ets__ -DICACHE_FLASH -DPLATFORMIO=030200 -DESP8266 -DARDUINO_ARCH_ESP8266 -DARDUINO_ESP8266_ESP12 -U__STRICT_ANSI__ -I/root/.platformio/packages/framework-arduinoespressif8266/tools/sdk/include -I/root/.platformio/packages/framework-arduinoespressif8266/tools/sdk/lwip/include -I/root/.platformio/packages/framework-arduinoespressif8266/cores/esp8266 -I/root/.platformio/packages/framework-arduinoespressif8266/variants/nodemcu /root/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/Tone.cpp
xtensa-lx106-elf-g++ -o .pioenvs/default/FrameworkArduino/Updater.o -c -fno-rtti -fno-exceptions -std=c++11 -Os -mlongcalls -mtext-section-literals -falign-functions=4 -ffunction-sections -fdata-sections -DARDUINO=20300 -DLWIP_OPEN_SRC -DF_CPU=160000000L -D__ets__ -DICACHE_FLASH -DPLATFORMIO=030200 -DESP8266 -DARDUINO_ARCH_ESP8266 -DARDUINO_ESP8266_ESP12 -U__STRICT_ANSI__ -I/root/.platformio/packages/framework-arduinoespressif8266/tools/sdk/include -I/root/.platformio/packages/framework-arduinoespressif8266/tools/sdk/lwip/include -I/root/.platformio/packages/framework-arduinoespressif8266/cores/esp8266 -I/root/.platformio/packages/framework-arduinoespressif8266/variants/nodemcu /root/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/Updater.cpp
/mnt/ramdrive/projects/MySONOFF-3569aba0d50347e8ed8362a338a2905eb8baddf5/src/MySONOFF.ino:21:26: fatal error: WemosArduino.h: No such file or directory
#include <WemosArduino.h>
^
compilation terminated.
*** [.pioenvs/default/src/MySONOFF.ino.o] Error 1
========================== [ERROR] Took 1.29 seconds ==========================

================================== [SUMMARY] ==================================
Environment default [ERROR]
Environment beta    [SKIP]
========================== [ERROR] Took 1.29 seconds ==========================

I can confirm that a simple sketch with no dependent libs in the /libs/ folder compiles just fine :)

A

sticilface commented 8 years ago

cache is working though! repeat compile goes from 12.5 seconds to 1.29!!!

ivankravets commented 8 years ago

its getting there, however, now it borks as libs included in the libs folder are not being copied.

It should copy. I've jsut re-tested this example

pio remote run -d atmelavr-and-arduino/arduino-external-libs -e uno -r -v
PlatformIO Plus (https://pioplus.com) v0.6.3
[Tue Nov 29 01:03:52 2016] Processing uno (platform: atmelavr, board: uno, framework: arduino)
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Collected 38 compatible libraries
Looking for dependencies...
Library Dependency Graph
|-- <OneWire> (/private/tmp/pioremote/projects/arduino-external-libs-6935f7980e9f3b10626236f68179a411c17f0ef9/lib/OneWire)
avr-g++ -o .pioenvs/uno/src/DS18x20_Temperature.o -c -fno-exceptions -fno-threadsafe-statics -fpermissive -std=gnu++11 -g -Os -Wall -ffunction-sections -fdata-sections -flto -mmcu=atmega328p -DF_CPU=16000000L -DPLATFORMIO=030200 -DARDUINO_ARCH_AVR -DARDUINO_AVR_UNO -DARDUINO=10612 -I/Users/ikravets/.platformio/packages/framework-arduinoavr/cores/arduino -I/Users/ikravets/.platformio/packages/framework-arduinoavr/variants/standard -Ilib/OneWire -Isrc src/DS18x20_Temperature.cpp
....

P.S: Do you mean .piolibdeps directory? We don't sync it because this is PlatformIO service fodler. Please don't modify it and use lib_deps option. PlatformIO will install all dependencies on remote side for the first build.

sticilface commented 8 years ago

mmm... so in the example above. it fails because #include <WemosArduino.h> is not there.

ah... i might have had this issue before... it doesn't follow symlinks does it?

ivankravets commented 8 years ago

ah... i might have had this issue before... it doesn't follow symlinks does it?

Maybe :) Will check it.

sticilface commented 8 years ago

yeah its the symlinks. i thought i'd copied in the files, but i hadn't. removing symlinks works like a charm!

ivankravets commented 8 years ago

Need to check it carefuly. I forgot to mention that I don't use RSYNC or etc tools. These tools work via open ports (SSH or etc) and requrie OS permissions.

I developed own "incremental sync" algorithm over PIO Remote protocol. As result, need to re-test all conditions. It works carzy fast. I even transferred 1Gb from A -> B :))) Thanks a lot that you help.

I like this PIO Remote technolodgy. It looks very interesting and safe. Need to find ways how to explain developers how to use it.

sticilface commented 8 years ago

it does look very cool indeed. lots of applications with cross-architecture compiling. if it can follow symlinks that would be awesome. i usually just link in my libs so that they stay update with different projects that I'm working on. :)

ivankravets commented 7 years ago

Please run pio upgrade/pio update to 0.6.4. Does it work now?

sticilface commented 7 years ago

:) works most excellently!!! thank you.

bit of a spanner.. tried to make this spiffs now it gets stuck even when i don't try the spiffs

Error: Could not find a version that satisfies the requirement '~1.102.0' for your system 'freebsd_amd64'

did you not make the spiffs tool for freebsd.. i thought we sorted that out at one point?

ivankravets commented 7 years ago

Thanks for report :)

Which package is missed?

sticilface commented 7 years ago

oops.. sorry thought i copied it. mkspiffs. 

PackageManager: Installing tool-mkspiffs @ ~1.102.0
Error: Could not find a version that satisfies the requirement '~1.102.0' for your system 'freebsd_amd64'

and you need it to upload something, regardless of if you are using uploadfs or just -t upload

ivankravets commented 7 years ago

Could you provide this package for FreeBSD?

Thanks!

sticilface commented 7 years ago

I've had a look at the makefile.. and i've genuinely not got a scoobie how to modify it to work with freebsd... i've asked ivan to take a look at modifying it for me so it can compile. a bit like the esptool....

https://github.com/igrr/mkspiffs/issues/16

The other thing might be to disable the mkspiffs requirement until you actually use uploadfs, so it doesn't get stuck!

ivankravets commented 7 years ago

Please download any file from here https://bintray.com/platformio/dl-packages/tool-mkspiffs/1.102.0#files

Unpack and change system to FreeBSD :)

sticilface commented 7 years ago

mkspiffs-0.1.2-Freebsd64.zip

ivankravets commented 7 years ago

Thanks a lot!!! I've already deployed it. It should work.

sticilface commented 7 years ago

boom. it does. thanks for the tip!

ivankravets commented 7 years ago

Please sorry, I didn't check. Is this binary compilied with FreeBSD?

sticilface commented 7 years ago

Yes!

On 3 Dec 2016, at 14:45, Ivan Kravets notifications@github.com wrote:

Please sorry, I didn't check. Is this binary compilied with FreeBSD?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

sticilface commented 7 years ago

and it works. i checked it.