search

ARMmbed / mbed-os

Arm Mbed OS is a platform operating system designed for the internet of things
https://mbed.com
Other
4.66k stars 2.97k forks source link

ARM_CM3DS_MPS2+ drive complains No space left on device #6460

Closed studavekar closed 6 years ago

studavekar commented 6 years ago

Description

Bug

Target ARM_CM3DS_MPS2

Toolchain: GCC_ARM|ARM|IAR

mbed-cli version: 1.5.0

mbed-os sha: d97d55a Merge pull request #6450 from 0xc0170/fix_#6449

Expected behavior During the execution of tests, the device storage should not be used up.

Actual behavior we see No space left on device error

Steps to reproduce Running tests in loop mbed test -m ARM_CM3DS_MPS2 -t GCC_ARM -v

Snippet of exception 

[1522080032.80][SERI][INF] serial(port=/dev/ttyUSB0, baudrate=9600, read_timeout=0.01, write_timeout=5)
[1522080032.80][SERI][INF] reset device using 'reboot.txt' plugin...
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/arm/repo/htrun/mbed_host_tests/host_tests_conn_proxy/conn_proxy.py", line 145, in conn_process
    connector = conn_primitive_factory(conn_resource, config, event_queue, logger)
  File "/home/arm/repo/htrun/mbed_host_tests/host_tests_conn_proxy/conn_proxy.py", line 105, in conn_primitive_factory
    config=config)
  File "/home/arm/repo/htrun/mbed_host_tests/host_tests_conn_proxy/conn_primitive_serial.py", line 75, in __init__
    self.reset_dev_via_serial(delay=self.forced_reset_timeout)
  File "/home/arm/repo/htrun/mbed_host_tests/host_tests_conn_proxy/conn_primitive_serial.py", line 92, in reset_dev_via_serial
    polling_timeout=self.config.get('polling_timeout'))
  File "/home/arm/repo/htrun/mbed_host_tests/host_tests_plugins/__init__.py", line 86, in call_plugin
    return HOST_TEST_PLUGIN_REGISTRY.call_plugin(type, capability, *args, **kwargs)
  File "/home/arm/repo/htrun/mbed_host_tests/host_tests_plugins/host_test_registry.py", line 69, in call_plugin
    return plugin.execute(capability, *args, **kwargs)
  File "/home/arm/repo/htrun/mbed_host_tests/host_tests_plugins/module_reset_mps2.py", line 83, in execute
    reboot_fh = open(reboot_file_path, "w")
IOError: [Errno 28] No space left on device: '/mnt/pci-0000_00_14_0-usb-0_7_1_0-scsi-0_0_0_0/reboot.txt'

arm@austin-ci-linux-002:/mnt$ df -i /mnt/pci-0000_00_14_0-usb-0_7_1_0-scsi-0_0_0_0 Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sdb1 0 0 0 - /mnt/pci-0000_00_14_0-usb-0_7_1_0-scsi-0_0_0_0


- this works by deleting a file on the drive

arm@austin-ci-linux-002:/mnt/pci-0000_00_14_0-usb-0_7_1_0-scsi-0_0_0_0$ touch reboot.txt touch: cannot touch 'reboot.txt': No space left on device

arm@austin-ci-linux-002:/mnt/pci-0000_00_14_0-usb-0_7_1_0-scsi-0_0_0_0$ ls config.txt MB mbed.elf mbed.htm SOFTWARE System Volume Information

arm@austin-ci-linux-002:/mnt/pci-0000_00_14_0-usb-0_7_1_0-scsi-0_0_0_0$ rm mbed.elf

arm@austin-ci-linux-002:/mnt/pci-0000_00_14_0-usb-0_7_1_0-scsi-0_0_0_0$ touch reboot.txt



cc : @ashok-rao @hug-dev
ashok-rao commented 6 years ago

@tkaman / @hug-dev ..could you please take a look at this?

hug-dev commented 6 years ago

Hi @studavekar I will look into this and try to see what we can do!

tkaman commented 6 years ago

Hi @studavekar, I could not reproduce the problem so far. How often did you see this? Only once or many times? What I experienced is that the D drive disconnected, and did not reconnected until I manually reseted the board. Did you experienced this issue, or just the no space left problem?

Which HT version are you using? I am using 1.3.1. As I see you are using Linux. Did you try this with Windows as well?

Anyway, this does not seem like a HAL issue, it is rather a MPS2 board or maybe an environment problem.

hug-dev commented 6 years ago

I could not reproduce this issue as well (having executed around 15 tests). The only thing I could see sometimes is that after copying the reboot.txt file and after the board reboots, it is not automounted by the OS anymore, I had to manually mount it or in the worst case reset the board manually (pushing the RESET button).

Configuration:

ashok-rao commented 6 years ago

@studavekar ..bump!

studavekar commented 6 years ago

@tkaman @hug-dev

Hi @studavekar, I could not reproduce the problem so far. How often did you see this? Only once or many times?

Once we see this problem I have no way to recover the device. Cold hot, unplugging sdcard doesn't help much.

What I experienced is that the D drive disconnected, and did not reconnected until I manually reseted the board. Did you experienced this issue, or just the no space left problem?

Yes i did see this issue, as you pointed out resetting device helps to recover.

Which HT version are you using? I am using 1.3.1.

yes 1.3.1

As I see you are using Linux. Did you try this with Windows as well?

I am using Linux.

Anyway, this does not seem like a HAL issue, it is rather a MPS2 board or maybe an environment problem.

@hug-dev

I could not reproduce this issue as well (having executed around 15 tests). This board was connected in staging CI, The number of tests it executed were of higher order.

on side one was on vacation and have few more devices let me try to reproduce it on other device.

studavekar commented 6 years ago

Did an overnight test with 2 boards didn't see No space left on device issue. However tests were stopped stalled because of device failed to reboot using reboot,txt.

tkaman commented 6 years ago

@ashok-rao , @studavekar Can we close this ticket now? Looks like the problem is seen only once, so we can't make progress on this.

For the reboot issue you can open a new ticket, although it should be investigated by the MPS2 team, as it does not look like a CM3DS issue. Let me know if you need anything from our side.

studavekar commented 6 years ago

closing this ticket as we are not able to reproduce this issue. Have opened https://github.com/ARMmbed/mbed-os/issues/6966 for tracking reboot.txt not been rebooting the device.