PowellTravis commented 4 years ago

This is autogenerated. Please review and update as needed.

Describe the bug

When I go to run any command on the system it errors out and prevents me from using the cloud shell interface.

Command Name Any command at all

Errors:

The command failed with an unexpected error. Here is the traceback:

[Errno 30] Read-only file system: '/home/**Redacted Username**/.azure/commands/2020-06-05.22-24-38.unknown_command.211.log'
Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 215, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 511, in execute
    load_cmd_tbl_func=self.commands_loader.load_command_table, args=args)
  File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 164, in raise_event
    func(self, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/azlogging.py", line 86, in init_command_file_logging
    self._init_command_logfile_handlers(cmd_logger, args)  # pylint: disable=protected-access
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/azlogging.py", line 108, in _init_command_logfile_handlers
    logfile_handler = logging.FileHandler(log_file_path)
  File "/opt/az/lib/python3.6/logging/__init__.py", line 1032, in __init__
    StreamHandler.__init__(self, self._open())
  File "/opt/az/lib/python3.6/logging/__init__.py", line 1061, in _open
    return open(self.baseFilename, self.mode, encoding=self.encoding)
OSError: [Errno 30] Read-only file system: '/home/**Redacted Username**/.azure/commands/2020-06-05.22-24-38.unknown_command.211.log'

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

I don't know how to reproduce the errors. I just get the error every time I put in a command.

Expected Behavior

I expected all the commands to not error out on run.

Environment Summary

Cloud Shell

Additional Context

ghost commented 4 years ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @maertendMSFT.

yonzhan commented 4 years ago

Cloud Shell

jiasli commented 4 years ago

This issue has been mentioned multiple times.

Looks like you somehow lost permission to /home/Redacted Username/.azure/commands/2020-06-05.22-24-38.unknown_command.211.log.

13848 is worse - even permission to `/home/username/.azure/az.sess` is lost:

OSError: [Errno 30] Read-only file system: '/home/username/.azure/az.sess'

Could you check the permission of this folder?

$ stat ~/.azure/commands
  File: '/home/leo/.azure/commands'
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: 700h/1792d      Inode: 131079      Links: 2
Access: (0755/drwxr-xr-x)  Uid: ( 9527/     leo)   Gid: ( 9527/     leo)
Access: 2020-06-09 08:11:16.161186717 +0000
Modify: 2020-06-09 08:10:56.481010326 +0000
Change: 2020-06-09 08:10:56.481010326 +0000
 Birth: -

$ ls ~/.azure/commands -l
total 92
-rw-r--r-- 1 leo leo 472 Jun  3 02:38 2020-06-03.02-38-06.ad_sp_credential_reset.694.log
-rw-r--r-- 1 leo leo 480 Jun  3 02:38 2020-06-03.02-38-13.ad_sp_credential_reset.739.log
-rw-r--r-- 1 leo leo 210 Jun  5 07:41 2020-06-05.07-41-35.login.103.log
-rw-r--r-- 1 leo leo 223 Jun  5 07:41 2020-06-05.07-41-38.account_set.116.log
-rw-r--r-- 1 leo leo 536 Jun  5 07:41 2020-06-05.07-41-43.rest.289.log

Also, could you try creating a file with

$ echo "Hello World" > ~/.azure/commands/test.txt

$ cat ~/.azure/commands/test.txt
Hello World

PowellTravis commented 4 years ago

I did the commands these were my outputs:

$ stat ~/.azure/commands
  File: '/home/**Redacted Username**/.azure/commands'
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: 700h/1792d      Inode: 131079      Links: 2
Access: (0755/drwxr-xr-x)  Uid: ( 9527/**Redacted Username**)   Gid: ( 9527/**Redacted Username**)
Access: 2020-06-09 14:38:02.020203476 +0000
Modify: 2020-06-09 14:38:01.504197941 +0000
Change: 2020-06-09 14:38:01.504197941 +0000
 Birth: -

$ ls ~/.azure/commands -l
total 32
-rw-r--r-- 1 **Redacted** **Redacted** 210 Jun  5 22:30 2020-06-05.22-30-36.login.115.log
-rw-r--r-- 1 **Redacted** **Redacted** 223 Jun  5 22:30 2020-06-05.22-30-39.account_set.126.log
-rw-r--r-- 1 **Redacted** **Redacted** 194 Jun  5 23:23 2020-06-05.23-23-58.unknown_command.183.log
-rw-r--r-- 1 **Redacted** **Redacted** 404 Jun  5 23:24 2020-06-05.23-24-12.aks_install-cli.190.log
-rw-r--r-- 1 **Redacted** **Redacted** 210 Jun  9 14:38 2020-06-09.14-37-59.login.103.log
-rw-r--r-- 1 **Redacted** **Redacted** 223 Jun  9 14:38 2020-06-09.14-38-01.account_set.115.log
-rw-r--r-- 1 **Redacted** **Redacted** 194 Jun  9 14:38 2020-06-09.14-38-50.unknown_command.140.log
-rw-r--r-- 1 **Redacted** **Redacted** 404 Jun  9 14:39 2020-06-09.14-39-19.aks_install-cli.145.log

Command with test.txt

$echo "Hello World" > ~/.azure/commands/test.txt
([Errno 13] Permission denied: '~/.azure/commands/test.txt')
$ cat ~/.azure/commands/test.txt
~/.azure/commands/test.txt does not exist

I had also tried to install az aks install-cli and I got this out

Downloading client to "/usr/local/bin/kubectl" from "https://storage.googleapis.com/kubernetes-release/release/v1.18.3/bin/linux/amd64/kubectl"
Connection error while attempting to download client ([Errno 13] Permission denied: '/usr/local/bin/kubectl')

smcgrawDotNet commented 4 years ago

Good day,

Different command; however, same error as above:

{OSError: [Errno 30] Read-only file system: '/home/samuel/.azure/az.sess'}

samuel@Azure:~$ echo "hello world" > ~/.azure/commands/test.txt bash: /home/samuel/.azure/commands/test.txt: Read-only file system

NOTE: Access was working until I deleted two 'Resource groups' via the gui. I am working from the 'cloud shell' in 'Edge'.

jiasli commented 4 years ago

Thanks for the information! I was wonder if you still have permission to the home folder. Could you try to write something to upper locations?

echo "hello world" > ~/.azure/test.txt
echo "hello world" > ~/test.txt

@PowellTravis, is **Redacted** yourself as shown in the terminal's prompt?

leo@Azure:~$

PowellTravis commented 4 years ago

Sorry for the late reply. Yes Redacted is myself.

jiasli commented 4 years ago

Hi @PowellTravis, how about the operations I mentioned above? Could you try to write to the home folder as well?

echo "hello world" > ~/.azure/test.txt
echo "hello world" > ~/test.txt

Benoitlotfallah commented 4 years ago

Hi @jiasli i have the same problem, and couldn't write on any of them

benoit@Azure:~$ echo "hello world" > ~/.azure/test.txt
bash: /home/benoit/.azure/test.txt: Read-only file system
benoit@Azure:~$ echo "hello world" > ~/test.txt
bash: /home/benoit/test.txt: Read-only file system

jiasli commented 4 years ago

Thanks @Benoitlotfallah for the information. We can isolate the issue to Cloud Shell itself. I will work with Cloud Shell team to see how we can address it.

edyoung commented 4 years ago

Unfortunately, it looks as though the file which stores your home directory may have become corrupted. However I don't know exactly how so we'll need to investigate a bit.

Can you (any user experiencing this problem) confirm:

does this issue persist across multiple Cloud Shell sessions?
if you are comfortable doing so, please share the value of "echo $ACC_PUID" - we can use this to identify the sessions you created and there might be information in the logs which will help

To work around the issue, you can delete and recreate your home directory, like this:

1) this will result in deleting any files which are under ~ apart from files which are in ~/clouddrive. Copy any files you need to keep under the ~/clouddrive directory

2) In the Cloud Shell toolbar, Click Upload/Download > Manage File Share

3) You'll see in the pane above Cloud Shell a view of your Cloud Drive file share, including a directory called .cloudconsole. Open that and you'll see a file called something like acc_username.img

4) In the Cloud Shell toolbar, click 'Restart Cloud Shell' and close the Cloud Shell pane before it starts a new shell. (This is so we unmount your home directory)

5) Delete the file acc_username.img.

6) Start Cloud Shell again. We will recreate a new home directory for you. The directory should be writable including the .azure directory. You can copy saved files from ~/clouddrive back into the home directory.

PowellTravis commented 4 years ago

This issue has persisted across all of my cloud shells, and kept up even when I delete the cloud shell storage and create a new one.

edyoung commented 4 years ago

@PowellTravis that's quite strange. Nothing is actually persisted outside the storage so I'm confused how that works.

When you say 'delete the cloud shell storage', do you mean actually delete the storage account or file share via the portal, or run 'clouddrive unmount'? If you run the latter and then reselect the same file share when you reconnect to cloud shell, you'd still be using the same files. Might be worth checking the date on that .img file I mention above.

PowellTravis commented 4 years ago

I have deleted the storage account where the cloud shell data is stored.

edyoung commented 4 years ago

Do you have write access to files in the ~/clouddrive directory? When you recreated the cloud drive storage after deleting it, did you see this 'access denied' issue immediately? If you try the steps I list above, does it make a difference? (If deleting the storage account didn't, it's unlikely)

PowellTravis commented 4 years ago

I do have access to the ~/clouddrive directory. That is the only thing I have the permission to view on the system still. I do get the access denied even after deleting and recreating the storage account.

Herve-Stephane commented 4 years ago

Hello,

Get the same error trying first to create AKS and now az login doesn't work anymore.

Stef

smcgrawDotNet commented 4 years ago

Good day,

Confirm (or fix) your login first. Make sure you have:

Close and open a new ACS tab;
Some-times ACS works better in Chrome than IE;
In both cases clear the browser cache (logout first - after logback-in);

Sam

On Tue, Jul 28, 2020 at 7:29 AM Herve-Stephane notifications@github.com wrote:

Hello,

Get the same error trying first to create AKS and now az login doesn't work anymore.

Stef

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Azure/azure-cli/issues/13855#issuecomment-665074580, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHNMBYFHDDXFQF2W3UIJR2DR53OFVANCNFSM4NVIQNGA .

edyoung commented 4 years ago

I don't believe this would be resolved by reopening Cloud Shell or using an alternative browser. Please see my comment earlier in this thread for troubleshooting steps.

Herve-Stephane commented 4 years ago

Hello, Thanks.

Sam it's the first thing i've done. Restarting everything. Ed i can't do it by command line but with the portal it works. I will try to recreate the directory as you explain.

Thanks again

Herve-Stephane commented 4 years ago

Hi,

This morning everything works fine. To be continued...

Herve-Stephane commented 4 years ago

Sorry in fact when i tried to write a file it works but nothing more.

maertendMSFT commented 4 years ago

@Herve-Stephane so you are only able to create a file, but you are still getting issues when running commands?

patricekrakow commented 4 years ago

Yes, I can reproduce the problem and a fix/workaround: 1/ Delete the group "cloud-shell-storage-westeurope" in which the "Storage account" for the Cloud Shell has been created. 2/ Wait for the deletion. 3/ Go back to Cloud Shell and click "Create storage". 4/ See the issue by typing (for instance):

$ echo "hello world" > ~/test.txt
bash: /home/benoit/test.txt: Read-only file system

5/ Restart Cloud Shell with the "power" button/icon. 6/ Check that the issue is fixed by typing (for instance):

$ echo "hello world" > ~/test.txt

Hope this help.

smcgrawDotNet commented 4 years ago

Good day,

Same problem -- I had to close and reopen any tools connected to Azure -- I was leaving my system locked at the end of the day, however, I would leave tools open.

On Thu, Nov 5, 2020 at 6:12 AM Patrice Krakow notifications@github.com wrote:

I had the same issue "Read-only file system", and it (might) has been fixed by restarting the cloud shell using the "power" button.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Azure/azure-cli/issues/13855#issuecomment-722402557, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHNMBYB4QR5PIP4VEHVUNTTSOKXDBANCNFSM4NVIQNGA .

maertendMSFT commented 3 years ago

@patricekrakow does this only occur the first time you are connecting to the new file share? My guess is that you are deleting the old file share, creating the new file share, then cloud shell is giving you the session before it has finished making the updates to the storage account settings for the new file share.

dempsey93 commented 3 years ago

Yes, I can reproduce the problem and a fix/workaround: 1/ Delete the group "cloud-shell-storage-westeurope" in which the "Storage account" for the Cloud Shell has been created. 2/ Wait for the deletion. 3/ Go back to Cloud Shell and click "Create storage". 4/ See the issue by typing (for instance):
$ echo "hello world" > ~/test.txt
bash: /home/benoit/test.txt: Read-only file system
5/ Restart Cloud Shell with the "power" button/icon. 6/ Check that the issue is fixed by typing (for instance):
$ echo "hello world" > ~/test.txt
Hope this help.

dempsey93 commented 3 years ago

It works perfectly, thanks a lot :)

balajiraman commented 3 years ago

Yes, I can reproduce the problem and a fix/workaround: 1/ Delete the group "cloud-shell-storage-westeurope" in which the "Storage account" for the Cloud Shell has been created. 2/ Wait for the deletion. 3/ Go back to Cloud Shell and click "Create storage". 4/ See the issue by typing (for instance):
$ echo "hello world" > ~/test.txt
bash: /home/benoit/test.txt: Read-only file system
5/ Restart Cloud Shell with the "power" button/icon. 6/ Check that the issue is fixed by typing (for instance):
$ echo "hello world" > ~/test.txt
Hope this help.

I have tried this, does not work.

When I deleted my storage account and recreated the shell.. its mounting a ready only drive. What I find using df command is that it does not mount he cloud shell file share. When I restart cloudshell, it hangs with black screen for ever, no error messages to be seen. I have done multiple times, with autogenerated CS namespaces to creating my own with no avail. is this specific user related issue with a hung process somewhere in the stack?

smcgrawDotNet commented 2 years ago

Mr. LI

Try this: Log-out of Auzer (don't just close the tab) Delete all the cookies in your browser(s) Log-back end

Sam

On Mon, Feb 7, 2022 at 10:03 PM Jiashuo Li @.***> wrote:

Reopened #13855 https://github.com/Azure/azure-cli/issues/13855.

— Reply to this email directly, view it on GitHub https://github.com/Azure/azure-cli/issues/13855#event-6024215393, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHNMBYHFHD5NEWMDM4IO2Y3U2CW2JANCNFSM4NVIQNGA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

suha-glal commented 2 years ago

I had the same issue but @dempsey93 solution solved it. Thanks you very much!

hacikho commented 2 years ago

Yes, I can reproduce the problem and a fix/workaround: 1/ Delete the group "cloud-shell-storage-westeurope" in which the "Storage account" for the Cloud Shell has been created. 2/ Wait for the deletion. 3/ Go back to Cloud Shell and click "Create storage". 4/ See the issue by typing (for instance):
$ echo "hello world" > ~/test.txt
bash: /home/benoit/test.txt: Read-only file system
5/ Restart Cloud Shell with the "power" button/icon. 6/ Check that the issue is fixed by typing (for instance):
$ echo "hello world" > ~/test.txt
Hope this help.

dude you are a life saver

jerrychong25 commented 11 months ago

Hi all,

I've faced similar error like your case, but inside Azure CLI via Jenkins pipeline.

Could share what is steps required on resolving this issue?

Error Logs:

23:14:49  + az account set --subscription ****
23:14:50  + az aks get-credentials --resource-group RG_1 --name AKS-1
23:14:54  ERROR: The command failed with an unexpected error. Here is the traceback:
23:14:54  ERROR: [Errno 30] Read-only file system: '/.kube/config'
23:14:54  Traceback (most recent call last):
23:14:54    File "/usr/lib64/az/lib/python3.9/site-packages/knack/cli.py", line 233, in invoke
23:14:54      cmd_result = self.invocation.execute(args)
23:14:54    File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/__init__.py", line 663, in execute
23:14:54      raise ex
23:14:54    File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
23:14:54      results.append(self._run_job(expanded_arg, cmd_copy))
23:14:54    File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/__init__.py", line 697, in _run_job
23:14:54      result = cmd_copy(params)
23:14:54    File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/__init__.py", line 333, in __call__
23:14:54      return self.handler(*args, **kwargs)
23:14:54    File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
23:14:54      return op(**command_args)
23:14:54    File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/command_modules/acs/custom.py", line 1280, in aks_get_credentials
23:14:54      _print_or_merge_credentials(
23:14:54    File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/command_modules/acs/custom.py", line 1403, in _print_or_merge_credentials
23:14:54      with os.fdopen(os.open(path, os.O_CREAT | os.O_WRONLY, 0o600), 'wt'):
23:14:54  OSError: [Errno 30] Read-only file system: '/.kube/config'
23:14:54  To check existing issues, please visit: https://github.com/Azure/azure-cli/issues

Kubctl Version Logs:

23:14:34  + az --version
23:14:36  azure-cli                         2.54.0
23:14:36  
23:14:36  core                              2.54.0
23:14:36  telemetry                          1.1.0
23:14:36  
23:14:36  Extensions:
23:14:36  azure-devops                      0.26.0
23:14:36  
23:14:36  Dependencies:
23:14:36  msal                            1.24.0b2
23:14:36  azure-mgmt-resource             23.1.0b2
23:14:36  
23:14:36  Python location '/usr/bin/python3.9'
23:14:36  Extensions directory '/opt/azcliextensions'
23:14:36  
23:14:36  Python (Linux) 3.9.18 (main, Sep 22 2023, 17:58:34) 
23:14:36  [GCC 8.5.0 20210514 (Red Hat 8.5.0-20)]
23:14:36  
23:14:36  Legal docs and information: aka.ms/AzureCliLegal
23:14:36  
23:14:36  
23:14:36  Your CLI is up-to-date.

Kubelogin Version Logs:

23:14:36  + kubelogin --version
23:14:36  kubelogin version 
23:14:36  git hash: v0.0.33/441bb556e8486866aa809e9c2b82397f8a01f364
23:14:36  Go version: go1.19.10
23:14:36  Build time: 2023-10-26T15:47:28Z
23:14:36  Platform: linux/amd64

Thank you.

Azure / azure-cli

[CloudShell] Error on every command run on Azure Cloud Shell: OSError: [Errno 30] Read-only file system #13855

This is autogenerated. Please review and update as needed.

Describe the bug

To Reproduce:

Expected Behavior

Environment Summary

Additional Context

13848 is worse - even permission to `/home/username/.azure/az.sess` is lost:

Azure / azure-cli

[CloudShell] Error on every command run on Azure Cloud Shell: OSError: [Errno 30] Read-only file system #13855

This is autogenerated. Please review and update as needed.

Describe the bug

To Reproduce:

Expected Behavior

Environment Summary

Additional Context

13848 is worse - even permission to /home/username/.azure/az.sess is lost:

13848 is worse - even permission to `/home/username/.azure/az.sess` is lost: