search

adafruit / circuitpython

CircuitPython - a Python implementation for teaching coding with microcontrollers
https://circuitpython.org
Other
4.11k stars 1.22k forks source link

Large Broadcom Ports Folder #5619

Closed matthewjerome closed 2 years ago

matthewjerome commented 2 years ago

CircuitPython version

7.0.0

Code/REPL

git submodule update --init

Behavior

When running git submodule update --init, the size is >8gb just for circuitpython/ports/broadcom/firmware

Is there a way to target a different repo or branch when building?

Long module sync time



### Description

_No response_

### Additional information

_No response_
matthewjerome commented 2 years ago

Build location has 20gb capacity :

Screen Shot 2021-11-25 at 10 05 14 PM
rpavlik commented 2 years ago

Probably the best approach is to just avoid initializing those large submodules that you don't need. I believe there may also be a submodule deinit command.

matthewjerome commented 2 years ago

Just ran git submodule deinit "ports/broadcom/peripherals" and git submodule deinit "ports/broadcom/firmware" which show the messages Cleared directory 'ports/broadcom/firmware' and Cleared directory 'ports/broadcom/peripherals'. After running git submodule update --init, the request for broadcom still happens. Tried with a fresh copy and then also removed references to broadcom in .gitmodules but then the error appears fatal: No url found for submodule path 'ports/broadcom/firmware' in .gitmodules.

rpavlik commented 2 years ago

Don't pass --init to the submodules update command, that initializes all currently used submodules, including the ones you just de-initted

jepler commented 2 years ago

I agree it would be nice to find a way so that the majority of developers, who do no build the broadcom port, can avoid this extremely time (during the initial clone / submodule init) and disk space intensive submodule (forever). If this means changing our documentation to not recommend git submodule update --init, so be it, but the 'correct' alternative will need to be short and easy to remember!

I investigated whether specifying that the submodule was shallow helped, but it appears not to. This included if I created a branch or a tag that pointed exactly at the desired ref, in a fork of rpi-firmware (with the submodule url changed to point to my fork).

These files are not "source files" in any traditional sense, and keeping their history is not useful. I wonder if Adafruit should create a fresh repo with the content of rpi-firmware but not the history, and point to this in gitmodules.

rpavlik commented 2 years ago

oh, it's the rpi stuff that's causing the issue. I was wondering what the broadcom ports folder was... Yeah, could do a fresh repo, or could use FetchContent.cmake to just download the files required from rpi-firmware. https://cmake.org/cmake/help/v3.13/module/FetchContent.html

dhalbert commented 2 years ago

I did git submodule update --init --depth 1 and that was much faster, though not instantaneous. We could recommend that, maybe?

dhalbert commented 2 years ago

I did git submodule update --init --depth 1 and that was much faster, though not instantaneous. We could recommend that, maybe?

@tannewt points out --depth 1` may not include the commit you want.

tannewt commented 2 years ago

Sorry about this folks. I'm a bit spoiled with my gigabit internet.

I've made a PR #5627 which will prevent the large download by setting shallow = true. This will fail, however, because the latest commit isn't the one we use. So, I added make fetch-submodules for the top level that does the specific fetch trick that the CI does. It should be used for updating instead of git submodule update directly.

RetiredWizard commented 2 years ago

Thanks Scott! It wasn't only the speed, I was pulling my hair out trying to free up space on my little build PIs, got up to 15G but still wasn't enough.

Anyway, I'm really clueless when it comes to make so I"m sure I'm doing something stupid. This is what I get when I try the make fetch-submodules:

make fetch-submodules BOARD=adafruit_feather_esp32s2 Use make V=1, make V=2 or set BUILD_VERBOSE similarly in your environment to increase build verbosity. find: ‘../../extmod/ulab/code’: No such file or directory make: *** No rule to make target 'fetch-submodules'. Stop.

Duh, nevermind, your PR hasn't been merged yet..... Time for bed :), I'll pick it up tomorrow. Thanks for taking care of this :D

matthewjerome commented 2 years ago

Just tried to build again - the build process is different but still has a huge size associated with it. Seems to fail around ports/broadcom/firmware. Take a look at the terminal output :

$ git submodule sync --quiet --recursive
$ git submodule update --init

Submodule 'data/nvm.toml' (https://github.com/adafruit/nvm.toml.git) registered for path 'data/nvm.toml'
Submodule 'extmod/ulab' (https://github.com/v923z/micropython-ulab) registered for path 'extmod/ulab'
Submodule 'frozen/Adafruit_CircuitPython_APDS9960' (https://github.com/adafruit/Adafruit_CircuitPython_APDS9960) registered for path 'frozen/Adafruit_CircuitPython_APDS9960'
Submodule 'frozen/Adafruit_CircuitPython_BLE' (https://github.com/adafruit/Adafruit_CircuitPython_BLE) registered for path 'frozen/Adafruit_CircuitPython_BLE'
Submodule 'frozen/Adafruit_CircuitPython_BLE_Apple_Notification_Center' (https://github.com/adafruit/Adafruit_CircuitPython_BLE_Apple_Notification_Center) registered for path 'frozen/Adafruit_CircuitPython_BLE_Apple_Notification_Center'
Submodule 'frozen/Adafruit_CircuitPython_BusDevice' (https://github.com/adafruit/Adafruit_CircuitPython_BusDevice.git) registered for path 'frozen/Adafruit_CircuitPython_BusDevice'
Submodule 'frozen/Adafruit_CircuitPython_CircuitPlayground' (https://github.com/adafruit/Adafruit_CircuitPython_CircuitPlayground.git) registered for path 'frozen/Adafruit_CircuitPython_CircuitPlayground'
Submodule 'frozen/Adafruit_CircuitPython_Crickit' (https://github.com/adafruit/Adafruit_CircuitPython_Crickit) registered for path 'frozen/Adafruit_CircuitPython_Crickit'
Submodule 'frozen/Adafruit_CircuitPython_DRV2605' (https://github.com/adafruit/Adafruit_CircuitPython_DRV2605) registered for path 'frozen/Adafruit_CircuitPython_DRV2605'
Submodule 'frozen/Adafruit_CircuitPython_DS3231' (https://github.com/adafruit/Adafruit_CircuitPython_DS3231) registered for path 'frozen/Adafruit_CircuitPython_DS3231'
Submodule 'frozen/Adafruit_CircuitPython_Display_Shapes' (https://github.com/adafruit/Adafruit_CircuitPython_Display_Shapes) registered for path 'frozen/Adafruit_CircuitPython_Display_Shapes'
Submodule 'frozen/Adafruit_CircuitPython_Display_Text' (https://github.com/adafruit/Adafruit_CircuitPython_Display_Text) registered for path 'frozen/Adafruit_CircuitPython_Display_Text'
Submodule 'frozen/Adafruit_CircuitPython_DotStar' (https://github.com/adafruit/Adafruit_CircuitPython_DotStar.git) registered for path 'frozen/Adafruit_CircuitPython_DotStar'
Submodule 'frozen/Adafruit_CircuitPython_ESP32SPI' (https://github.com/adafruit/Adafruit_CircuitPython_ESP32SPI) registered for path 'frozen/Adafruit_CircuitPython_ESP32SPI'
Submodule 'frozen/Adafruit_CircuitPython_FocalTouch' (https://github.com/adafruit/Adafruit_CircuitPython_FocalTouch) registered for path 'frozen/Adafruit_CircuitPython_FocalTouch'
Submodule 'frozen/Adafruit_CircuitPython_HID' (https://github.com/adafruit/Adafruit_CircuitPython_HID.git) registered for path 'frozen/Adafruit_CircuitPython_HID'
Submodule 'frozen/Adafruit_CircuitPython_IRRemote' (https://github.com/adafruit/Adafruit_CircuitPython_IRRemote.git) registered for path 'frozen/Adafruit_CircuitPython_IRRemote'
Submodule 'frozen/Adafruit_CircuitPython_LC709203F' (https://github.com/adafruit/Adafruit_CircuitPython_LC709203F) registered for path 'frozen/Adafruit_CircuitPython_LC709203F'
Submodule 'frozen/Adafruit_CircuitPython_LIS3DH' (https://github.com/adafruit/Adafruit_CircuitPython_LIS3DH.git) registered for path 'frozen/Adafruit_CircuitPython_LIS3DH'
Submodule 'frozen/Adafruit_CircuitPython_LSM6DS' (https://github.com/adafruit/Adafruit_CircuitPython_LSM6DS) registered for path 'frozen/Adafruit_CircuitPython_LSM6DS'
Submodule 'frozen/Adafruit_CircuitPython_MIDI' (https://github.com/adafruit/Adafruit_CircuitPython_MIDI) registered for path 'frozen/Adafruit_CircuitPython_MIDI'
Submodule 'ports/atmel-samd/Adafruit_CircuitPython_Motor' (https://github.com/adafruit/Adafruit_CircuitPython_Motor.git) registered for path 'frozen/Adafruit_CircuitPython_Motor'
Submodule 'atmel-samd/frozen/Adafruit_CircuitPython_NeoPixel' (https://github.com/adafruit/Adafruit_CircuitPython_NeoPixel) registered for path 'frozen/Adafruit_CircuitPython_NeoPixel'
Submodule 'frozen/Adafruit_CircuitPython_ProgressBar' (https://github.com/adafruit/Adafruit_CircuitPython_ProgressBar) registered for path 'frozen/Adafruit_CircuitPython_ProgressBar'
Submodule 'frozen/Adafruit_CircuitPython_RFM69' (https://github.com/adafruit/Adafruit_CircuitPython_RFM69.git) registered for path 'frozen/Adafruit_CircuitPython_RFM69'
Submodule 'frozen/Adafruit_CircuitPython_RFM9x' (https://github.com/adafruit/Adafruit_CircuitPython_RFM9x.git) registered for path 'frozen/Adafruit_CircuitPython_RFM9x'
Submodule 'frozen/Adafruit_CircuitPython_Register' (https://github.com/adafruit/Adafruit_CircuitPython_Register.git) registered for path 'frozen/Adafruit_CircuitPython_Register'
Submodule 'frozen/Adafruit_CircuitPython_Requests' (https://github.com/adafruit/Adafruit_CircuitPython_Requests) registered for path 'frozen/Adafruit_CircuitPython_Requests'
Submodule 'frozen/Adafruit_CircuitPython_SD' (https://github.com/adafruit/Adafruit_CircuitPython_SD.git) registered for path 'frozen/Adafruit_CircuitPython_SD'
Submodule 'frozen/Adafruit_CircuitPython_ST7789' (https://github.com/adafruit/Adafruit_CircuitPython_ST7789) registered for path 'frozen/Adafruit_CircuitPython_ST7789'
Submodule 'frozen/Adafruit_CircuitPython_SimpleIO' (https://github.com/adafruit/adafruit_circuitpython_simpleio) registered for path 'frozen/Adafruit_CircuitPython_SimpleIO'
Submodule 'frozen/Adafruit_CircuitPython_SimpleMath' (https://github.com/adafruit/Adafruit_CircuitPython_SimpleMath) registered for path 'frozen/Adafruit_CircuitPython_SimpleMath'
Submodule 'frozen/Adafruit_CircuitPython_Thermistor' (https://github.com/adafruit/Adafruit_CircuitPython_Thermistor.git) registered for path 'frozen/Adafruit_CircuitPython_Thermistor'
Submodule 'ports/atmel-samd/Adafruit_CircuitPython_seesaw' (https://github.com/adafruit/Adafruit_CircuitPython_seesaw.git) registered for path 'frozen/Adafruit_CircuitPython_seesaw'
Submodule 'frozen/circuitpython-stage' (https://github.com/python-ugame/circuitpython-stage.git) registered for path 'frozen/circuitpython-stage'
Submodule 'frozen/pew-pewpew-standalone-10.x' (https://github.com/pewpew-game/pew-pewpew-standalone-10.x.git) registered for path 'frozen/pew-pewpew-standalone-10.x'
Submodule 'lib/axtls' (https://github.com/micropython/axtls.git) registered for path 'lib/axtls'
Submodule 'lib/berkeley-db-1.xx' (https://github.com/pfalcon/berkeley-db-1.xx) registered for path 'lib/berkeley-db-1.xx'
Submodule 'lib/libffi' (https://github.com/atgreen/libffi) registered for path 'lib/libffi'
Submodule 'lib/mp3' (https://github.com/adafruit/Adafruit_MP3) registered for path 'lib/mp3'
Submodule 'lib/nrfutil' (https://github.com/adafruit/nRF52_nrfutil) registered for path 'lib/nrfutil'
Submodule 'lib/protomatter' (https://github.com/adafruit/Adafruit_Protomatter) registered for path 'lib/protomatter'
Submodule 'lib/quirc' (https://github.com/adafruit/quirc.git) registered for path 'lib/quirc'
Submodule 'lib/tinyusb' (https://github.com/hathach/tinyusb.git) registered for path 'lib/tinyusb'
Submodule 'atmel-samd/asf4' (https://github.com/adafruit/asf4.git) registered for path 'ports/atmel-samd/asf4'
Submodule 'ports/atmel-samd/freetouch' (https://github.com/adafruit/Adafruit_FreeTouch.git) registered for path 'ports/atmel-samd/freetouch'
Submodule 'ports/atmel-samd/peripherals' (https://github.com/adafruit/samd-peripherals.git) registered for path 'ports/atmel-samd/peripherals'
Submodule 'rpi-firmware' (https://github.com/raspberrypi/rpi-firmware.git) registered for path 'ports/broadcom/firmware'
Submodule 'ports/broadcom/peripherals' (https://github.com/adafruit/broadcom-peripherals.git) registered for path 'ports/broadcom/peripherals'
Submodule 'ports/cxd56/spresense-exported-sdk' (https://github.com/sonydevworld/spresense-exported-sdk.git) registered for path 'ports/cxd56/spresense-exported-sdk'
Submodule 'ports/espressif/certificates/nina-fw' (https://github.com/adafruit/nina-fw.git) registered for path 'ports/espressif/certificates/nina-fw'
Submodule 'ports/espressif/esp-idf' (https://github.com/espressif/esp-idf.git) registered for path 'ports/espressif/esp-idf'
Submodule 'ports/mimxrt10xx/sdk' (https://github.com/adafruit/MIMXRT10xx_SDK) registered for path 'ports/mimxrt10xx/sdk'
Submodule 'ports/nrf/nrfx' (https://github.com/adafruit/nrfx.git) registered for path 'ports/nrf/nrfx'
Submodule 'ports/raspberrypi/sdk' (https://github.com/raspberrypi/pico-sdk.git) registered for path 'ports/raspberrypi/sdk'
Submodule 'ports/stm/st_driver' (https://github.com/hathach/st_driver.git) registered for path 'ports/stm/st_driver'
Submodule 'tools/Tecate-bitmap-fonts' (https://github.com/Tecate/bitmap-fonts.git) registered for path 'tools/Tecate-bitmap-fonts'
Submodule 'tools/adabot' (https://github.com/adafruit/adabot.git) registered for path 'tools/adabot'
Submodule 'tools/bitmap_font' (https://github.com/adafruit/Adafruit_CircuitPython_BitmapFont.git) registered for path 'tools/bitmap_font'
Submodule 'tools/huffman' (https://github.com/tannewt/huffman.git) registered for path 'tools/huffman'
Submodule 'tools/python-semver' (https://github.com/k-bx/python-semver.git) registered for path 'tools/python-semver'
Submodule 'tools/uf2' (https://github.com/Microsoft/uf2.git) registered for path 'tools/uf2'
Submodule 'tools/usb_descriptor' (https://github.com/adafruit/usb_descriptor.git) registered for path 'tools/usb_descriptor'
Cloning into '/Volumes/circuitpy/circuitpython/data/nvm.toml'...
Cloning into '/Volumes/circuitpy/circuitpython/extmod/ulab'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_APDS9960'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_BLE'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_BLE_Apple_Notification_Center'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_BusDevice'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_CircuitPlayground'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_Crickit'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_DRV2605'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_DS3231'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_Display_Shapes'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_Display_Text'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_DotStar'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_ESP32SPI'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_FocalTouch'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_HID'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_IRRemote'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_LC709203F'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_LIS3DH'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_LSM6DS'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_MIDI'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_Motor'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_NeoPixel'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_ProgressBar'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_RFM69'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_RFM9x'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_Register'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_Requests'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_SD'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_ST7789'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_SimpleIO'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_SimpleMath'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_Thermistor'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/Adafruit_CircuitPython_seesaw'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/circuitpython-stage'...
Cloning into '/Volumes/circuitpy/circuitpython/frozen/pew-pewpew-standalone-10.x'...
Cloning into '/Volumes/circuitpy/circuitpython/lib/axtls'...
Cloning into '/Volumes/circuitpy/circuitpython/lib/berkeley-db-1.xx'...
Cloning into '/Volumes/circuitpy/circuitpython/lib/libffi'...
Cloning into '/Volumes/circuitpy/circuitpython/lib/mp3'...
Cloning into '/Volumes/circuitpy/circuitpython/lib/nrfutil'...
Cloning into '/Volumes/circuitpy/circuitpython/lib/protomatter'...
Cloning into '/Volumes/circuitpy/circuitpython/lib/quirc'...
Cloning into '/Volumes/circuitpy/circuitpython/lib/tinyusb'...
Cloning into '/Volumes/circuitpy/circuitpython/ports/atmel-samd/asf4'...
Cloning into '/Volumes/circuitpy/circuitpython/ports/atmel-samd/freetouch'...
Cloning into '/Volumes/circuitpy/circuitpython/ports/atmel-samd/peripherals'...
Cloning into '/Volumes/circuitpy/circuitpython/ports/broadcom/firmware'...
Cloning into '/Volumes/circuitpy/circuitpython/ports/broadcom/peripherals'...
Cloning into '/Volumes/circuitpy/circuitpython/ports/cxd56/spresense-exported-sdk'...
Cloning into '/Volumes/circuitpy/circuitpython/ports/espressif/certificates/nina-fw'...
Cloning into '/Volumes/circuitpy/circuitpython/ports/espressif/esp-idf'...
Cloning into '/Volumes/circuitpy/circuitpython/ports/mimxrt10xx/sdk'...
Cloning into '/Volumes/circuitpy/circuitpython/ports/nrf/nrfx'...
Cloning into '/Volumes/circuitpy/circuitpython/ports/raspberrypi/sdk'...
Cloning into '/Volumes/circuitpy/circuitpython/ports/stm/st_driver'...
Cloning into '/Volumes/circuitpy/circuitpython/tools/Tecate-bitmap-fonts'...
Cloning into '/Volumes/circuitpy/circuitpython/tools/adabot'...
Cloning into '/Volumes/circuitpy/circuitpython/tools/bitmap_font'...
Cloning into '/Volumes/circuitpy/circuitpython/tools/huffman'...
Cloning into '/Volumes/circuitpy/circuitpython/tools/python-semver'...
Cloning into '/Volumes/circuitpy/circuitpython/tools/uf2'...
Cloning into '/Volumes/circuitpy/circuitpython/tools/usb_descriptor'...
Submodule path 'data/nvm.toml': checked out 'e5b149599d14a8841167fe552846ca36925b87a0'
Submodule path 'extmod/ulab': checked out 'b913d064e525f674d0219524988e6d9d834fe09c'
Submodule path 'frozen/Adafruit_CircuitPython_APDS9960': checked out 'c55da0dee66302d2fa8ed31623d047c307f409b2'
Submodule path 'frozen/Adafruit_CircuitPython_BLE': checked out '781f46b73f71817b709d2e279fc188a7c1e69423'
Submodule path 'frozen/Adafruit_CircuitPython_BLE_Apple_Notification_Center': checked out 'c24a5310bc259cd9d93b1f42468e07f41d4b0e56'
Submodule path 'frozen/Adafruit_CircuitPython_BusDevice': checked out '169715b3444c614e55827ccf79b35b2b5e11f1d2'
Submodule path 'frozen/Adafruit_CircuitPython_CircuitPlayground': checked out '2017afdfb43d3d9c5a73f8e85e951a583b18206a'
Submodule path 'frozen/Adafruit_CircuitPython_Crickit': checked out '8d09b29a1a92499dbbd10dd832f27db71057af5f'
Submodule path 'frozen/Adafruit_CircuitPython_DRV2605': checked out '8a6ab89b7d19f45a20b3f794cf900e23c9a8453b'
Submodule path 'frozen/Adafruit_CircuitPython_DS3231': checked out '92a6ce1da70c0c4f69ef34fc0d7ec7155cc2441f'
Submodule path 'frozen/Adafruit_CircuitPython_Display_Shapes': checked out '313c17cbb590ef8d505845369eb3b78c25c97fae'
Submodule path 'frozen/Adafruit_CircuitPython_Display_Text': checked out 'ca141954c0b39cc0343c63bedb64c02f307159b9'
Submodule path 'frozen/Adafruit_CircuitPython_DotStar': checked out 'e07090117766d4a9ea2de07cd6f5418990cc598b'
Submodule path 'frozen/Adafruit_CircuitPython_ESP32SPI': checked out 'e25c1801c5428598f543873bb846851b78efb329'
Submodule path 'frozen/Adafruit_CircuitPython_FocalTouch': checked out '68744ede79c992a3df8322c21a1468c5ccaef2ee'
Submodule path 'frozen/Adafruit_CircuitPython_HID': checked out 'd79dd180cf6062e97d6a12cbc8dc7fdbedcc752b'
Submodule path 'frozen/Adafruit_CircuitPython_IRRemote': checked out '2ca37f927b3ee3aad379c2991f36b3ef1be0203d'
Submodule path 'frozen/Adafruit_CircuitPython_LC709203F': checked out 'ed41dd44605110f1d1411ed694ec2b5676dd8d9c'
Submodule path 'frozen/Adafruit_CircuitPython_LIS3DH': checked out 'daff8cdca78637915bea2f3a120cc84578516195'
Submodule path 'frozen/Adafruit_CircuitPython_LSM6DS': checked out '9873446ebde5779c03c2bd4e3e544a801fb27bd5'
Submodule path 'frozen/Adafruit_CircuitPython_MIDI': checked out '1127e3f7bcefa9fddb5b7f30533ecc6c58b420ea'
Submodule path 'frozen/Adafruit_CircuitPython_Motor': checked out '88e5ae51427aa690de129c157223e93bf936ac41'
Submodule path 'frozen/Adafruit_CircuitPython_NeoPixel': checked out '9ca3bf00c6a2dd1de2d315a3b952a381d720aa7d'
Submodule path 'frozen/Adafruit_CircuitPython_ProgressBar': checked out 'e4f7cd083dea8eb103954e7703bbb487491b433f'
Submodule path 'frozen/Adafruit_CircuitPython_RFM69': checked out '70ef86dc1a42138efcab01945d5a421e45f2408e'
Submodule path 'frozen/Adafruit_CircuitPython_RFM9x': checked out '379209dd7e4374c262b84256778d7895693e7fb6'
Submodule path 'frozen/Adafruit_CircuitPython_Register': checked out '79678c6adb2252de8fed6273604bc6ac676132a5'
Submodule path 'frozen/Adafruit_CircuitPython_Requests': checked out '900b28cbae008e3253c4c40496e49faea9fb7034'
Submodule path 'frozen/Adafruit_CircuitPython_SD': checked out 'b7a76420d1dec119f8744aa7c0ea500e235561d1'
Submodule path 'frozen/Adafruit_CircuitPython_ST7789': checked out 'c5b480434de8fa56d8ba978a57cd3919fdc9da2a'
Submodule path 'frozen/Adafruit_CircuitPython_SimpleIO': checked out 'ca56187fe7af315130808191b004432fdfdc1b09'
Submodule path 'frozen/Adafruit_CircuitPython_SimpleMath': checked out '5b699a609197fb42aa059788c7c824c346d57c4e'
Submodule path 'frozen/Adafruit_CircuitPython_Thermistor': checked out 'd4ac6ce3eea2c87781fa2df4e431d9440c610fad'
Submodule path 'frozen/Adafruit_CircuitPython_seesaw': checked out '0450119e163df782930a887022e197d6d25ca666'
Submodule path 'frozen/circuitpython-stage': checked out '2c89a329063848bcd7dba6af6fb6c9d6036dd4f2'
Submodule path 'frozen/pew-pewpew-standalone-10.x': checked out '88b8be84b5dce7660f58c02a63263f1d2ff0709f'
Submodule path 'lib/axtls': checked out '531cab9c278c947d268bd4c94ecab9153a961b43'
Submodule path 'lib/berkeley-db-1.xx': checked out '35aaec4418ad78628a3b935885dd189d41ce779b'
Submodule path 'lib/libffi': checked out 'e9de7e35f2339598b16cbb375f9992643ed81209'
Submodule path 'lib/mp3': checked out '7a5de1ad777e95b0f4fab7bbd35678c7d319b1b5'
Submodule path 'lib/nrfutil': checked out '9e7dfb28a5c6f3d7a19340971b32e0c2b4128ecf'
Submodule path 'lib/protomatter': checked out '7fe6406affb1376193102cf76ded06e61316d7e6'
Submodule path 'lib/quirc': checked out '8c6ffa228a4c7643daed7039d3c51d38a43991b8'
Submodule path 'lib/tinyusb': checked out 'ae73873b5cba0eb11c89165f4559964940430d44'
Submodule path 'ports/atmel-samd/asf4': checked out '84f56af13292d8f32c40acbd949bde698ddd4507'
Submodule path 'ports/atmel-samd/freetouch': checked out 'b6859a349efb79e2008d6f6a34e406e93a8e19c0'
Submodule path 'ports/atmel-samd/peripherals': checked out 'd3b20192cf94fdea68a412596e082108ba5ebbf0'
remote: Enumerating objects: 1783364, done.
remote: Counting objects: 100% (1783361/1783361), done.
remote: Compressing objects: 100% (196993/196993), done.
fatal: write error: No space left on device8.22 GiB | 10.20 MiB/s
Fetched in submodule path 'ports/broadcom/firmware', but it did not contain 6a5207946edcd45813d1dd1572ca8bd8101b68b6. Direct fetching of that commit failed.
dhalbert commented 2 years ago

Do not do this anymore:

$ git submodule sync --quiet --recursive
$ git submodule update --init

Instead, do this:

$ cd circuitpython     # go to top level
$ make fetch-submodules

That does something trickier, which fetches only as much of a submodule as needed. This was added in #5627, which closed this issue.

The https://learn.adafruit.com/building-circuitpython guide has been updated to reflect this.

jerryneedell commented 2 years ago

After doing a clean clone and the make fetch-submodules I get this with git diff Did miss something?

git clone https://github.com/adafruit/circuitpython.git
cd circuitpython
make fetch-submodules

git diff
diff --git a/ports/broadcom/firmware b/ports/broadcom/firmware
--- a/ports/broadcom/firmware
+++ b/ports/broadcom/firmware
@@ -1 +1 @@
-Subproject commit 6a5207946edcd45813d1dd1572ca8bd8101b68b6
+Subproject commit 6a5207946edcd45813d1dd1572ca8bd8101b68b6-dirty
jerryneedell commented 2 years ago

Hmmm -- I reran this on my linux box and did not see the diff .... never-mind...

jerryneedell commented 2 years ago

very odd -- this is reproducible on the Mac, but not under Linux (Ubuntu 20.04)

Does anyone know how to fix it on the Mac?

tannewt commented 2 years ago

@jerryneedell what does git status show in the firmware directory and at the top level?

dhalbert commented 2 years ago

Also what are the versions of git you are using?

jerryneedell commented 2 years ago 
jerryneedell@Mac-mini firmware % git status
HEAD detached at 6a52079
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
    modified:   modules/5.10.81+/kernel/net/netfilter/xt_DSCP.ko
    modified:   modules/5.10.81+/kernel/net/netfilter/xt_HL.ko
    modified:   modules/5.10.81+/kernel/net/netfilter/xt_RATEEST.ko
    modified:   modules/5.10.81+/kernel/net/netfilter/xt_TCPMSS.ko
    modified:   modules/5.10.81-v7+/kernel/net/netfilter/xt_DSCP.ko
    modified:   modules/5.10.81-v7+/kernel/net/netfilter/xt_HL.ko
    modified:   modules/5.10.81-v7+/kernel/net/netfilter/xt_RATEEST.ko
    modified:   modules/5.10.81-v7+/kernel/net/netfilter/xt_TCPMSS.ko
    modified:   modules/5.10.81-v7l+/kernel/net/netfilter/xt_DSCP.ko
    modified:   modules/5.10.81-v7l+/kernel/net/netfilter/xt_HL.ko
    modified:   modules/5.10.81-v7l+/kernel/net/netfilter/xt_RATEEST.ko
    modified:   modules/5.10.81-v7l+/kernel/net/netfilter/xt_TCPMSS.ko
    modified:   modules/5.10.81-v8+/kernel/net/netfilter/xt_DSCP.ko
    modified:   modules/5.10.81-v8+/kernel/net/netfilter/xt_HL.ko
    modified:   modules/5.10.81-v8+/kernel/net/netfilter/xt_RATEEST.ko
    modified:   modules/5.10.81-v8+/kernel/net/netfilter/xt_TCPMSS.ko

no changes added to commit (use "git add" and/or "git commit -a")
jerryneedell@Mac-mini firmware %
jerryneedell commented 2 years ago

jerryneedell@Mac-mini firmware % git --version git version 2.31.1 jerryneedell@Mac-mini firmware %

jerryneedell commented 2 years ago

that was on Mac

on Linux 

jerryneedell@jerryneedell-ubuntu-macmini:~/projects/circuitpython/ports/broadcom$ cd firmware/
jerryneedell@jerryneedell-ubuntu-macmini:~/projects/circuitpython/ports/broadcom/firmware$ git status
HEAD detached at 6a52079
nothing to commit, working tree clean
jerryneedell@jerryneedell-ubuntu-macmini:~/projects/circuitpython/ports/broadcom/firmware$ git --version
git version 2.25.1
tannewt commented 2 years ago

I wonder if we need to do some sort of clean after checking out the correct commit. Files from the missing commit may not be deleted on checkout of a different commit.

matthewjerome commented 2 years ago

using an older version of git

$ which git
/usr/local/bin/git

$ git --version
git version 2.23.0