Which version of the blobfuse was used?

amd64 1.4.2 latest Get:1 https://packages.microsoft.com/repos/microsoft-ubuntu-bionic-prod bionic/main amd64 blobfuse amd64 1.4.2 [7437 kB]

Which OS (please include version) are you using?

Ubuntu 18.04.6 LTS

What problem was encountered?

On Azure HDInsight headnode, mount Azure storage account with blobfuse.: Following below document, and write a shell script check whether mount paths are still alive or not, getting following error:

Shell Script:

_check_blob() { i=" /datagrid/nfs_pool /datagrid/nas_pool /parallelrun /datagrid/outbound /datagrid/cde /datagrid/calc /datagrid/dgo /datagrid/partnerget /datagrid/scbcon /datagrid/dynasty /datagrid/dgp /datagrid/dip_hms /datagrid/cra /datagrid/dss /datagrid/cloud_pool /datagrid/sfmc /datagrid/rbh /datagrid/cloud_pool/aws/rbh /datagrid/pointx /datagrid/cloud_pool/gcp/pointx /datagrid/rdt /datagrid/dlprod_log /datagrid/dlprod_config/config /datagrid/dlprod /datagrid/cloud_pool/sfmc/sfmc /datagrid/dlprod/data_prep "

for PT in $i do out=`ps -ef |grep blobfuse | grep -v grep | awk -F' ' '{print $9}' | grep "$PT$"` if [[ $PT != $out ]] then echo "$PT :: blob not found" fi done }

Error: /datagrid/cloud_pool/aws/rbh :: blob not found

Ref

https://docs.microsoft.com/en-us/azure/storage/blobs/storage-how-to-mount-container-linux

Mount command: sudo blobfuse ~/mycontainer --tmp-path=/mnt/resource/blobfusetmp --config-file=/path/to/fuse_connection.cfg -o attr_timeout=240 -o entry_timeout=240 -o negative_timeout=120

Have you found a mitigation/solution?

By default, blobfuse logs errors to syslog. If this is relevant, is there anything in the syslog that might be helpful?

If relevant, please share your mount command.

using Azure HDInsight, On HDInsight headnode, mount Azure storage account with blobfuse.: sudo blobfuse ~/mycontainer --tmp-path=/mnt/resource/blobfusetmp --config-file=/path/to/fuse_connection.cfg -o attr_timeout=240 -o entry_timeout=240 -o negative_timeout=120

Sorry could not understand the exact issue here. From above text I assume after hitting mount command you are trying to check if blobfuse binary is running or not (through script) and that step fails. Can you mount without 'sudo' and try the script. If you suspect that blobfuse binary is going down after mounting, kindly enable LOG_DEBUG while mounting and share those logs.

Hi vibhansa, Thanks a lot for your quick reply.

yes, my provided log shows blob cannnot found. Seems no mount error as I understand. I set --log-level=LOG_DEBUG in the mount command and wait for the issue reoccurrency and will send you the mount logs.

Could you elaborate on why donot use sudo? Is it related to os user permission problem? If yes, kindly explain more.

Regards

for 'sudo' I wans just suspecting may be blobfuse was executed with 'sudo' while the script is running in normal user mode hence creating some issue. I went through the text again and I assume there are multiple mounts done on the system and script is checking for all mount points, where one is not found.

Hi Vibhansa, Thanks for your explanation. yeah, script is checking for all mount points, but only one is not found. So it's so strange. Except setting --log-level=LOG_DEBUG, do we have any other suggestion to investigate further?

No collecting debug logs shall be fine, we just need to make sure why the mount goes down. One other way to try out it is just use mount command for that particular path and config and check manually if it works or not (instead of checking through script). If there is any auth error or something, it will make it clear with this test case.

Hi Vibhansa, Thanks for your quick response. This issue is waiting for reproduce. Once it reproduced, I will send you the mount log for diaglositic.

Thanks

Hi Vibhansa， This issue happened again. And got the mount log. But the size is so huge. I just run grep error and found some exception. Could you help diagnostic if it's related to the issue.

Jan 13 03:11:40 xxxxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/b2k/log_b2k_2022-01-12_P6170_20220113_100131.log because of an error from upload_file_to_blob(). Errno = 400. Jan 13 03:11:40 xxxxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/frc/log_frc_2022-01-12_P9982_20220113_030753.log because of an error from upload_file_to_blob(). Errno = 400. Jan 13 04:09:50 xxxxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/frc/log_frc_2022-01-12_P9982_20220113_030753.log because of an error from upload_file_to_blob(). Errno = 400. Jan 13 04:18:12 xxxxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/common/inv_dly/pms_prod_prfl_dly_2022-01-12_20220113_110056.log because of an error from upload_file_to_blob(). Errno = 400. Jan 13 04:18:12 xxxxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/common/inv_dly/pms_cust_prod_dly_2022-01-12_20220113_110055.log because of an error from upload_file_to_blob(). Errno = 400. Jan 13 06:02:43 xxxxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/leads/log_leads_2022-01-12_P3762_20220113_125838.log because of an error from upload_file_to_blob(). Errno = 400. Jan 13 06:23:35 xxxxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/im/log_im_2022-01-12_P32499_20220113_125824.log because of an error from upload_file_to_blob(). Errno = 400. Jan 13 06:49:58 xxxxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/im/log_IM_2022-01-12_P3027_20220113_133437.log because of an error from upload_file_to_blob(). Errno = 400. Jan 13 09:45:28 xxxxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/win/log_win_2022-01-11_P1482_20220112_145458.log because of an error from upload_file_to_blob(). Errno = 400. Jan 13 10:20:48 xxxxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/leads/log_leads_2022-01-12_P11645_20220113_171803.log because of an error from upload_file_to_blob(). Errno = 400. Jan 13 10:46:41 xxxxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/leads/log_leads_2022-01-12_P14475_20220113_173922.log because of an error from upload_file_to_blob(). Errno = 400. Jan 13 12:50:15 xxxxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/win/log_win_2022-01-12_P12098_20220113_164737.log because of an error from upload_file_to_blob(). Errno = 400. Jan 13 15:10:12 xxxxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/imexi/log_imexi_2022-01-12_P18295_20220113_215846.log because of an error from upload_file_to_blob(). Errno = 400. Jan 13 15:52:16 xxxxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/s1/log_s1_2022-01-12_P2208_20220113_222626.log because of an error from upload_file_to_blob(). Errno = 400. Jan 13 16:13:11 xxxxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/s1/log_s1_2022-01-12_P25948_20220113_231245.log because of an error from upload_file_to_blob(). Errno = 400. Jan 13 16:26:38 xxxxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/win/log_win_2022-01-12_P3075_20220113_164654.log because of an error from upload_file_to_blob(). Errno = 400.

Jan 13 22:24:31 xxxxxxxxx blobfuse[1818]: Function azs_getattr, in file /home/vsts/work/1/s/blobfuse/utilities.cpp, line 193: Object /mnt/azureblobtmp/dlprod/root/raw/etl/common/bin/persist_audit/ats_error_online_itmx_trans_1_0.hql is not in the local cache during getattr. Jan 13 22:24:31 xxxxxxxxx blobfuse[1818]: ==> REQUEST/RESPONSE :: GET https://xxxxxxxxx.blob.core.windows.net/framework-hdi40?comp=list&delimiter=/&include=metadata&maxresults=5000&prefix=raw/etl/common/bin/persist_audit/ats_error_online_itmx_trans_1_0.hql&restype=container&sp=rwdl&st=2021-03-12T09:07:10Z&se=2099-12-31T09:07:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Thu, 13 Jan 2022 22:24:31 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 200 :: REQ ID : 89d0dadf-f01e-0010-7fcc-080870000000 Jan 13 22:24:31 xxxxxxxxx blobfuse[1818]: Function azs_getattr, in file /home/vsts/work/1/s/blobfuse/utilities.cpp, line 257: In azs_getattr list_segmented_item do loop blob prefix: raw/etl/common/bin/persist_audit/ats_error_online_itmx_trans_1_0.hql continuation: Jan 13 22:24:31 xxxxxxxxx blobfuse[1818]: Function azs_getattr, in file /home/vsts/work/1/s/blobfuse/utilities.cpp, line 278: In azs_getattr list_segmented_item 0 file raw/etl/common/bin/persist_audit/ats_error_online_itmx_trans_1_0.hql Jan 13 22:24:31 xxxxxxxxx blobfuse[1818]: Function azs_getattr, in file /home/vsts/work/1/s/blobfuse/utilities.cpp, line 301: In azs_getattr found blob in list file raw/etl/common/bin/persist_audit/ats_error_online_itmx_trans_1_0.hql Jan 13 22:24:31 xxxxxxxxx blobfuse[1818]: Function azs_getattr, in file /home/vsts/work/1/s/blobfuse/utilities.cpp, line 340: Blob /raw/etl/common/bin/persist_audit/ats_error_online_itmx_trans_1_0.hql, representing a file, found during get_attr. Jan 13 22:24:31 xxxxxxxxx blobfuse[1818]: azs_open called with path = /raw/etl/common/bin/persist_audit/ats_error_online_itmx_trans_1_0.hql, fi->flags = 8000. Jan 13 22:24:31 xxxxxxxxx blobfuse[1818]: Starting download of file /raw/etl/common/bin/persist_audit/ats_error_online_itmx_trans_1_0.hql Jan 13 22:24:31 xxxxxxxxx blobfuse[1818]: ==> REQUEST/RESPONSE :: GET https://xxxxxxxxx.blob.core.windows.net/framework-hdi40/raw/etl/common/bin/persist_audit/ats_error_online_itmx_trans_1_0.hql?sp=rwdl&st=2021-03-12T09:07:10Z&se=2099-12-31T09:07:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&x-ms-range=bytes=0-16777215&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Thu, 13 Jan 2022 22:24:31 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 206 :: REQ ID : 01524332-c01e-0031-58cc-086541000000 Jan 13 22:24:31 xxxxxxxxx blobfuse[1818]: Successfully downloaded blob raw/etl/common/bin/persist_audit/ats_error_online_itmx_trans_1_0.hql into file cache as /mnt/azureblobtmp/dlprod/root/raw/etl/common/bin/persist_audit/ats_error_online_itmx_trans_1_0.hql. Jan 13 22:24:31 xxxxxxxxx blobfuse[1818]: Function azs_open, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 202: Opening /mnt/azureblobtmp/dlprod/root/raw/etl/common/bin/persist_audit/ats_error_online_itmx_trans_1_0.hql gives fh = 73, errno = 0 Jan 13 22:24:31 xxxxxxxxx blobfuse[1818]: Function azs_open, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 237: Returning success from azs_open, file = /raw/etl/common/bin/persist_audit/ats_error_online_itmx_trans_1_0.hql Jan 13 22:24:31 xxxxxxxxx blobfuse[1818]: Function azs_flush, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 363: azs_flush called with path = /raw/etl/common/bin/persist_audit/ats_error_online_itmx_trans_1_0.hql, fi->flags = 0, (((struct fhwrapper )fi->fh)->fh) = 73. Jan 13 22:24:31 xxxxxxxxx blobfuse[1818]: Function azs_flush, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 404: azs_flush: path = /raw/etl/common/bin/persist_audit/ats_error_online_itmx_trans_1_0.hql, upload_on_close = 0, modified only flag config_options.uploadIfModified = 0, . Jan 13 22:25:26 xxxxxxxxx blobfuse[1818]: Function azs_flush, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 363: azs_flush called with path = /raw/etl/common/bin/persist_audit/ats_error_online_itmx_trans_1_0.hql, fi->flags = 0, (((struct fhwrapper )fi->fh)->fh) = 73. Jan 13 22:25:26 xxxxxxxxx blobfuse[1818]: Function azs_flush, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 404: azs_flush: path = /raw/etl/common/bin/persist_audit/ats_error_online_itmx_trans_1_0.hql, upload_on_close = 0, modified only flag config_options.uploadIfModified = 0, . Jan 13 22:25:26 xxxxxxxxx blobfuse[1818]: Function azs_release, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 473: azs_release called with path = /raw/etl/common/bin/persist_audit/ats_error_online_itmx_trans_1_0.hql, fi->flags = 32768 Jan 13 22:25:26 xxxxxxxxx blobfuse[1818]: Function azs_release, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 502: Adding file to the GC from azs_release. File = /mnt/azureblobtmp/dlprod/root/raw/etl/common/bin/persist_audit/ats_error_online_itmx_trans_1_0.hql#012. Jan 13 22:25:28 xxxxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/common/collect_stat/log_2022-01-11_20220114_052528.log because of an error from upload_file_to_blob(). Errno = 400. Jan 13 22:27:27 xxxxxxxxx blobfuse[1818]: Function run_gc_cache, in file /home/vsts/work/1/s/blobfuse/src/BlobfuseGcCache.cpp, line 160: File /raw/etl/common/bin/persist_audit/ats_error_online_itmx_trans_1_0.hql being considered for deletion by file cache GC. Jan 13 22:27:27 xxxxxxxxx blobfuse[1818]: Function run_gc_cache, in file /home/vsts/work/1/s/blobfuse/src/BlobfuseGcCache.cpp, line 229: File /raw/etl/common/bin/persist_audit/ats_error_online_itmx_trans_1_0.hql deleted successfully by file cache GC. Jan 13 22:33:47 xxxxxxxxx blobfuse[1818]: Function azs_getattr, in file /home/vsts/work/1/s/blobfuse/utilities.cpp, line 120: azs_getattr called with path = /raw/etl/ats/src/hv_tbl_ats_error_online_trans_1_0.hql

Jan 13 22:25:28 ed11-hdi08p blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/common/collect_stat/log_2022-01-11_20220114_052528.log because of an error from upload_file_to_blob(). Errno = 400.

Uploads are failing due to client error. Can you share the PUT request from the log file.

Hi vibhansa, Thanks for your quick reply. Share with you the period log

Jan 18 04:12:40 xxxxxxx blobfuse[1818]: Function azs_readdir, in file /home/vsts/work/1/s/blobfuse/directoryapis.cpp, line 149: azs_readdir : About to call list_blobs. Container = framework-hdi40, delimiter = /, continuation = , prefix = raw/etl/common/bin/ Jan 18 04:12:40 xxxxxxx blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://storageaccount.blob.core.windows.net/framework-log/202201/raw/etl/common/inv_dly/pms_cust_dly_2022-01-17_20220118_110029.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=77314&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Tue, 18 Jan 2022 04:12:40 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: Jan 18 04:12:40 xxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/common/inv_dly/pms_cust_dly_2022-01-17_20220118_110029.log because of an error from upload_file_to_blob(). Errno = 400. Jan 18 04:12:40 xxxxxxx blobfuse[1818]: Function azs_getattr, in file /home/vsts/work/1/s/blobfuse/utilities.cpp, line 120: azs_getattr called with path = /raw/etl/common/bin

@Baiys1234 : Thanks for sharing those logs. In logs posted above I see there was only one failure (with status code 400), rest all put were successful. REQUEST/RESPONSE :: PUT https://xxxxxxxxx.blob.core.windows.net/framework-log/202201/raw/etl/common/collect_stat/log_2022-01-11_20220114_052528.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=867&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Thu, 13 Jan 2022 22:25:28 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 ::

And only this file has the path "framework-log/202201/raw/etl/common", rest are on a different path. What kind of auth you are using like account-key/SAS. Can you check auth that you are using is valid for this path as well. As per logs this particular upload was done to a different container.

Hi Vibhansa, I am using the Azure HDInsight cluster with Enterprise Security Package(ESP) Please find below doc regarding ESP details on HDInsight

https://docs.microsoft.com/en-us/azure/hdinsight/domain-joined/hdinsight-security-overview

Authentication Enterprise Security Package from HDInsight provides Active Directory-based authentication, multi-user support, and role-based access control. The Active Directory integration is achieved through the use of Azure Active Directory Domain Services. With these capabilities, you can create an HDInsight cluster joined to an Active Directory domain. Then configure a list of employees from the enterprise who can authenticate to the cluster.

With this setup, enterprise employees can sign in to the cluster nodes by using their domain credentials. They can also use their domain credentials to authenticate with other approved endpoints. Like Apache Ambari Views, ODBC, JDBC, PowerShell, and REST APIs to interact with the cluster.

Yesterday, this issue also happened again: Not only time, but serveral times. After that, no job ran successfully after 1/18 7:32 UTC time.

Jan 18 03:24:34 xxxxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/b2k/log_b2k_2022-01-17_P8405_20220118_101815.log because of an error from upload_file_to_blob(). Errno = 400. Jan 18 03:54:53 xxxxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/leads/log_leads_2022-01-17_P26396_20220118_104622.log because of an error from upload_file_to_blob(). Errno = 400. Jan 18 04:02:54 xxxxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/common/area_log/log_inv_dly_2022-01-17_20220118_110020.log because of an error from upload_file_to_blob(). Errno = 400. Jan 18 04:12:40 xxxxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/common/inv_dly/pms_cust_dly_2022-01-17_20220118_110029.log because of an error from upload_file_to_blob(). Errno = 400. Jan 18 04:30:52 xxxxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/common/bcm_cust_dly/cust_acct_tmp_bcm_dly_2022-01-17_20220118_110020.log because of an error from upload_file_to_blob(). Errno = 400. Jan 18 06:12:12 xxxxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/common/area_log/log_bcm_cmmrc_acct_dly_2022-01-16_20220118_130027.log because of an error from upload_file_to_blob(). Errno = 400. Jan 18 06:31:17 xxxxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/im/log_im_2022-01-17_P19111_20220118_131236.log because of an error from upload_file_to_blob(). Errno = 400. Jan 18 06:35:46 xxxxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/leads/log_leads_2022-01-17_P25320_20220118_131450.log because of an error from upload_file_to_blob(). Errno = 400. Jan 18 06:36:00 xxxxxxxxx blobfuse[1818]: Function azs_getattr, in file /home/vsts/work/1/s/blobfuse/utilities.cpp, line 120: azs_getattr called with path = /support_scripts/get_inputoutput_error.sh Jan 18 06:36:00 xxxxxxxxx blobfuse[1818]: Function azs_getattr, in file /home/vsts/work/1/s/blobfuse/utilities.cpp, line 193: Object /mnt/azureblobtmp/dlprod/root/support_scripts/get_inputoutput_error.sh is not in the local cache during getattr. Jan 18 06:36:00 xxxxxxxxx blobfuse[1818]: ==> REQUEST/RESPONSE :: GET https://scbdipseasta005stdprd.blob.core.windows.net/framework-hdi40?comp=list&delimiter=/&include=metadata&maxresults=5000&prefix=support_scripts/get_inputoutput_error.sh&restype=container&sp=rwdl&st=2021-03-12T09:07:10Z&se=2099-12-31T09:07:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Tue, 18 Jan 2022 06:36:00 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 200 :: REQ ID : 0ec119f4-401e-010d-6a35-0c97cf000000 Jan 18 06:36:00 xxxxxxxxx blobfuse[1818]: Function azs_getattr, in file /home/vsts/work/1/s/blobfuse/utilities.cpp, line 257: In azs_getattr list_segmented_item do loop blob prefix: support_scripts/get_inputoutput_error.sh continuation:

I was asking more about what kind of auth you are using to connect to container and not the HDInsight auth. You will have a config file for blobfuse, can you share that (redact the account name and key if any).

My doubt is in some flow you are trying to access a different container and there might be some issue with that mount. Jan 13 22:25:28 xxxxxxxxx blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://xxxxxxxxx.blob.core.windows.net/framework-log/202201/raw/etl/common/collect_stat/log_2022-01-11_20220114_052528.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=867&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Thu, 13 Jan 2022 22:25:28 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 ::

If you see this log pid of blobfuse is 3964 while for others its 1818. Also the container name is different here. So you need to check the mount config and commands for "framework-log" container. If you see all the logs that you have been posting only this particular instance is having issue and returning back with err 400.

Hi Vibhansa, Cx is referring below doc to mount storage to HDInsight.

https://docs.microsoft.com/en-us/azure/storage/blobs/storage-how-to-mount-container-linux

So it's just storageaccesskey confirmed from Azure storage account.

accountName myaccount accountKey storageaccesskey containerName mycontainer

Ok so you are using accountKey then it should not have caused any issue. From blobfuse side there is no failure if you see the above logs. Error code is coming from the server itself. Let me see if from server we can get more info on this.

In the logs where you got response code 400, can you look at the log file and find out there will be a token after that looking something like "d9d6c6a5-c01e-009b-47cc-08b3ae000000", we need that token to search the server logs.

sure， let me search.

I also confirmed many error like this.

Jan 18 03:24:34 XXXXXXXX blobfuse[1818]: Function azs_write, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 352: azs_write called with path= /raw/preprocinput/sfdc_inc/sedel1QEI Jan 18 03:24:34 XXXXXXXX blobfuse[1818]: message repeated 5 times: [ Function azs_write, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 352: azs_write called with path= /raw/preprocinput/sfdc_inc/sedel1QEI] Jan 18 03:24:34 XXXXXXXX blobfuse[1818]: Function azs_release, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 473: azs_release called with path = /raw/etl/common/bin/module/hive3_wrapper.sh, fi->flags = 32800 Jan 18 03:24:34 XXXXXXXX blobfuse[1818]: Function azs_release, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 502: Adding file to the GC from azs_release. File = /mnt/azureblobtmp/dlprod/root/raw/etl/common/bin/module/hive3_wrapper.sh#012. Jan 18 03:24:34 XXXXXXXX blobfuse[1818]: Function azs_write, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 352: azs_write called with path= /raw/preprocinput/sfdc_inc/sedel1QEI Jan 18 03:24:34 XXXXXXXX blobfuse[1818]: message repeated 14 times: [ Function azs_write, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 352: azs_write called with path= /raw/preprocinput/sfdc_inc/sedel1QEI] Jan 18 03:24:34 XXXXXXXX blobfuse[1818]: azs_open called with path = /raw/etl/common/bin/module/hive3_wrapper.sh, fi->flags = 8000. Jan 18 03:24:34 XXXXXXXX blobfuse[1818]: Function azs_write, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 352: azs_write called with path= /raw/preprocinput/sfdc_inc/sedel1QEI Jan 18 03:24:34 XXXXXXXX blobfuse[1818]: Function azs_open, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 202: Opening /mnt/azureblobtmp/dlprod/root/raw/etl/common/bin/module/hive3_wrapper.sh gives fh = 117, errno = 0 Jan 18 03:24:34 XXXXXXXX blobfuse[1818]: Function azs_open, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 237: Returning success from azs_open, file = /raw/etl/common/bin/module/hive3_wrapper.sh Jan 18 03:24:34 XXXXXXXX blobfuse[1818]: Function azs_write, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 352: azs_write called with path= /raw/preprocinput/sfdc_inc/sedel1QEI Jan 18 03:24:34 XXXXXXXX blobfuse[1818]: Function azs_write, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 352: azs_write called with path= /raw/preprocinput/sfdc_inc/sedel1QEI Jan 18 03:24:34 XXXXXXXX blobfuse[1818]: Function azs_flush, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 363: azs_flush called with path = /raw/etl/common/bin/module/hive3_wrapper.sh, fi->flags = 0, (((struct fhwrapper )fi->fh)->fh) = 117. Jan 18 03:24:34 XXXXXXXX blobfuse[1818]: Function azs_flush, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 404: azs_flush: path = /raw/etl/common/bin/module/hive3_wrapper.sh, upload_on_close = 0, modified only flag config_options.uploadIfModified = 0, . Jan 18 03:24:34 XXXXXXXX blobfuse[1818]: Function azs_write, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 352: azs_write called with path= /raw/preprocinput/sfdc_inc/sedel1QEI Jan 18 03:24:34 XXXXXXXX blobfuse[1818]: message repeated 8 times: [ Function azs_write, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 352: azs_write called with path= /raw/preprocinput/sfdc_inc/sedel1QEI] Jan 18 03:24:34 XXXXXXXX blobfuse[1818]: Function azs_flush, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 363: azs_flush called with path = /raw/etl/common/bin/module/hive3_wrapper.sh, fi->flags = 0, (((struct fhwrapper )fi->fh)->fh) = 117. Jan 18 03:24:34 XXXXXXXX blobfuse[1818]: Function azs_flush, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 404: azs_flush: path = /raw/etl/common/bin/module/hive3_wrapper.sh, upload_on_close = 0, modified only flag config_options.uploadIfModified = 0, . Jan 18 03:24:34 XXXXXXXX blobfuse[1818]: Function azs_write, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 352: azs_write called with path= /raw/preprocinput/sfdc_inc/sedel1QEI Jan 18 03:24:34 XXXXXXXX blobfuse[1818]: message repeated 711 times: [ Function azs_write, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 352: azs_write called with path= /raw/preprocinput/sfdc_inc/sedel1QEI] Jan 18 03:24:34 XXXXXXXX blobfuse[1818]: Function azs_write, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 352: azs_write called with path= /raw/preprocinput/sfdc_inc/sedel1QEI Jan 18 03:24:34 XXXXXXXX blobfuse[1818]: message repeated 32 times: [ Function azs_write, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 352: azs_write called with path= /raw/preprocinput/sfdc_inc/sedel1QEI] Jan 18 03:24:34 XXXXXXXX blobfuse[1818]: Function azs_write, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 352: azs_write called with path= /raw/preprocinput/sfdc_inc/sedel1QEI Jan 18 03:24:34 XXXXXXXX blobfuse[1818]: message repeated 552 times: [ Function azs_write, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 352: azs_write called with path= /raw/preprocinput/sfdc_inc/sedel1QEI] Jan 18 03:24:34 XXXXXXXX blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/b2k/log_b2k_2022-01-17_P8405_20220118_101815.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=33180&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Tue, 18 Jan 2022 03:24:34 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 665ed1f5-f01e-0054-491a-0cd41c000000 Jan 18 03:24:34 XXXXXXXX blobfuse[1818]: Function azs_write, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 352: azs_write called with path= /raw/preprocinput/sfdc_inc/sedel1QEI Jan 18 03:24:34 XXXXXXXX blobfuse[1818]: Function azs_write, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 352: azs_write called with path= /raw/preprocinput/sfdc_inc/sedel1QEI Jan 18 03:24:34 xxxxxxxxx blobfuse[3964]: Failing blob upload in azs_flush with input path /202201/raw/etl/b2k/log_b2k_2022-01-17_P8405_20220118_101815.log because of an error from upload_file_to_blob(). Errno = 400. Jan 18 03:24:34 xxxxxxxxx blobfuse[1818]: Function azs_write, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 352: azs_write called with path= /raw/preprocinput/sfdc_inc/sedel1QEI Jan 18 03:24:35 xxxxxxxxx blobfuse[1818]: message repeated 4640 times: [ Function azs_write, in file /home/vsts/work/1/s/blobfuse/fileapis.cpp, line 352: azs_write called with path= /raw/preprocinput/sfdc_inc/sedel1QEI] Jan 18 03:24:35 xxxxxxxxx blobfuse[1818]: Function run_gc_cache, in file /home/vsts/work/1/s/blobfuse/src/BlobfuseGcCache.cpp, line 160: File /raw/etl/common/bin/.sqlcmd_tmp/tmp_5e6ea43dbf425adcd0eb351864408b25_1642476153904790214.sql being considered for deletion by file cache GC. Jan 18 03:24:35 xxxxxxxxx blobfuse[1818]: Function run_gc_cache, in file /home/vsts/work/1/s/blobfuse/src/BlobfuseGcCache.cpp, line 160: File /raw/etl/common/bin/.sqlcmd_tmp/tmp_406b9b175e654f311bbf918e541905df_1642476154027215812.sql being considered for deletion by file cache GC.

Regarding "d9d6c6a5-c01e-009b-47cc-08b3ae000000", I searched all the logs and find below.

Jan 13 22:25:28 XXXXXXXX blobfuse[1818]: ==> REQUEST/RESPONSE :: DELETE https://scbdipseasta005stdprd.blob.core.windows.net/framework-hdi40/raw/etl/common/bin/.sqlcmd_tmp/tmp_6a073ed2ef1b6e039d29b8e95fd15f3e_1642112728522439463.sql?sp=rwdl&st=2021-03-12T09:07:10Z&se=2099-12-31T09:07:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&x-ms-delete-snapshots=include&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Thu, 13 Jan 2022 22:25:28 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 202 :: REQ ID : d9d6c6a5-c01e-009b-47cc-08b3ae000000

this is a valid request which was successful, do you see similar id string when response status was :: 400 ::

No, didnot see any 400 around Jan 13 22:25:28. But I saw many 400 error. and shared with you perviousely. So may I understand this issue is related to server side? the server side means Azure storage side?

here is what I searched:

blobfuse-rest.log-20220113"(137052,535):Jan 12 02:24:51 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/am/log_am_2022-01-11_P29209_20220112_092451.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=3129&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Wed, 12 Jan 2022 02:24:51 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 97dd11bd-a01e-0031-3f5b-07827b000000 blobfuse-rest.log-20220113"(447936,536):Jan 12 06:11:52 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/im/log_im_2022-01-11_P13468_20220112_130340.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=43413&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Wed, 12 Jan 2022 06:11:52 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 497de1bc-601e-00a4-177b-07e348000000 blobfuse-rest.log-20220113"(530323,539):Jan 12 07:23:02 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/sfdc/log_sfdc_2022-01-11_P8368_20220112_142134.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=34005&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Wed, 12 Jan 2022 07:23:02 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 12c86956-901e-001b-2685-07f73e000000 blobfuse-rest.log-20220113"(686509,553):Jan 12 10:11:05 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/common/area_log/log_bcm_dbcr_dly_2022-01-10_20220112_171019.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=108685&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Wed, 12 Jan 2022 10:11:05 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 0e970aec-901e-0093-4b9c-074fe7000000 blobfuse-rest.log-20220113"(1096653,542):Jan 12 16:32:07 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/imexi/log_imexi_2022-01-11_P28383_20220112_232401.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=39105&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Wed, 12 Jan 2022 16:32:07 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 5dfaf635-a01e-0021-39d1-0753a7000000 blobfuse-rest.log-20220113"(1099105,562):Jan 12 16:34:09 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/common/area_log/log_ads_prod_crcard_txn_d_2022-01-10_20220112_232118.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=235917&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Wed, 12 Jan 2022 16:34:09 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 69e3da8a-101e-007c-5ed2-07a3a3000000 blobfuse-rest.log-20220113"(1170488,557):Jan 12 18:01:32 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/common/area_log/log_lunar_prod_bcm_d_2022-01-10_20220113_010032.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=130618&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Wed, 12 Jan 2022 18:01:32 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 38320f18-f01e-00ba-0cde-07de9f000000 blobfuse-rest.log-20220113"(1170489,557):Jan 12 18:01:32 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/common/area_log/log_lunar_prod_bcm_d_2022-01-10_20220113_010032.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=130618&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Wed, 12 Jan 2022 18:01:32 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 12cf379b-101e-0055-1fde-07d5e1000000 blobfuse-rest.log-20220113"(1170490,557):Jan 12 18:01:32 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/common/area_log/log_lunar_prod_bcm_d_2022-01-10_20220113_010032.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=130752&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Wed, 12 Jan 2022 18:01:32 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 0c7f30da-e01e-00e1-17de-07d9e3000000 blobfuse-rest.log-20220113"(1170552,540):Jan 12 18:01:34 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/sba/log_sba_2022-01-11_P11901_20220112_142142.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=3736467&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Wed, 12 Jan 2022 18:01:34 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 96b71160-201e-013f-57de-07cf1f000000 blobfuse-rest.log-20220113"(1170553,557):Jan 12 18:01:34 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/common/area_log/log_lunar_prod_bcm_d_2022-01-10_20220113_010032.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=144551&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Wed, 12 Jan 2022 18:01:34 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 28aeb0f1-801e-0079-70de-0757dc000000 blobfuse-rest.log-20220113"(1170554,542):Jan 12 18:01:34 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/imexi/log_imexi_2022-01-11_P22547_20220113_005823.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=24317&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Wed, 12 Jan 2022 18:01:34 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 24a6d5e4-801e-0050-36de-07219e000000 blobfuse-rest.log-20220113"(1170779,557):Jan 12 18:01:43 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/common/area_log/log_lunar_prod_bcm_d_2022-01-10_20220113_010032.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=200325&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Wed, 12 Jan 2022 18:01:43 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : a201bd1d-f01e-0076-38de-07ba2a000000 blobfuse-rest.log-20220113"(1170780,540):Jan 12 18:01:43 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/sba/log_sba_2022-01-11_P11901_20220112_142142.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=3736467&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Wed, 12 Jan 2022 18:01:43 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 7b031d45-401e-00c5-41de-0740ad000000 blobfuse-rest.log-20220113"(1183416,538):Jan 12 18:13:46 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/mps/log_mps_2022-01-12_P20788_20220113_010248.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=86042&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Wed, 12 Jan 2022 18:13:46 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : cbcc8bfe-201e-0030-3ee0-0764bc000000 blobfuse-rest.log-20220113"(1235471,539):Jan 12 19:11:44 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/inv/log_inv_2022-01-12_P26209_20220113_004417.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=587740&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Wed, 12 Jan 2022 19:11:44 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : bfbbea5e-e01e-0026-0ee8-07a522000000 blobfuse-rest.log-20220113"(1252720,534):Jan 12 19:31:25 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/s1/log_s1_2022-01-11_P7719_20220113_022826.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=8848&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Wed, 12 Jan 2022 19:31:25 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 8f1ad2f7-001e-000e-6cea-07d29d000000 blobfuse-rest.log-20220113"(1252722,534):Jan 12 19:31:25 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/s1/log_s1_2022-01-11_P7719_20220113_022826.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=8848&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Wed, 12 Jan 2022 19:31:25 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : dac46608-601e-0051-51ea-072063000000 blobfuse-rest.log-20220113"(1282497,549):Jan 12 20:13:06 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/common/area_log/log_scbs_dly_2022-01-11_20220113_031018.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=213170&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Wed, 12 Jan 2022 20:13:06 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 785cb866-d01e-0025-0ff0-07a625000000 blobfuse-rest.log-20220113"(1282498,571):Jan 12 20:13:06 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/lunar/lunar_dw_prod_bcm_d/transform/log_dim_bcm_epp_org_info_2022-01-10_20220113_031247.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=9842&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Wed, 12 Jan 2022 20:13:06 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : f921cf38-a01e-0107-39f0-078e46000000 blobfuse-rest.log-20220113"(1344949,559):Jan 12 21:30:58 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/common/area_log/log_ci_cust_basic_excls_2022-01-11_20220113_043017.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=91368&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Wed, 12 Jan 2022 21:30:58 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 714f6213-401e-00e7-34fb-072e9b000000 blobfuse-rest.log-20220114"(373813,536):Jan 13 06:49:58 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/im/log_IM_2022-01-12_P3027_20220113_133437.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=125560&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Thu, 13 Jan 2022 06:49:58 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 1c9a54c3-101e-00dd-0f49-086d38000000 blobfuse-rest.log-20220114"(529767,539):Jan 13 09:45:28 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/win/log_win_2022-01-11_P1482_20220112_145458.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=8085206&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Thu, 13 Jan 2022 09:45:28 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 4b875d5f-901e-00a1-1562-08f00d000000 blobfuse-rest.log-20220114"(871880,542):Jan 13 15:10:12 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/imexi/log_imexi_2022-01-12_P18295_20220113_215846.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=36574&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Thu, 13 Jan 2022 15:10:12 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 23ba0007-201e-005d-348f-08ce92000000 blobfuse-rest.log-20220114"(949837,539):Jan 13 16:26:38 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/win/log_win_2022-01-12_P3075_20220113_164654.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=2229927&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Thu, 13 Jan 2022 16:26:38 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 663f922f-f01e-0076-309a-08ba2a000000 blobfuse-rest.log-20220114"(956566,542):Jan 13 16:32:28 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/imexi/log_imexi_2022-01-12_P30282_20220113_232440.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=83975&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Thu, 13 Jan 2022 16:32:28 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : e05ba8c2-101e-013c-189b-08cc18000000 blobfuse-rest.log-20220114"(1029678,557):Jan 13 18:01:54 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/common/area_log/log_dash_sale_dail_2022-01-11_20220113_190051.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=12909672&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Thu, 13 Jan 2022 18:01:54 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 741a8341-c01e-013e-08a7-08cee2000000 blobfuse-rest.log-20220114"(1029680,575):Jan 13 18:01:54 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/common/lunar_prod_bcm_d/lunar_fact_bcm_epp_doc_dtl_dly_d_2022-01-11_20220114_010057.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=2832&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Thu, 13 Jan 2022 18:01:54 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 36989b3a-001e-0027-07a7-08a4df000000 blobfuse-rest.log-20220114"(1029682,537):Jan 13 18:01:54 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/sfc/log_sfc_2022-01-13_P14194_20220114_010112.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=4974&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Thu, 13 Jan 2022 18:01:54 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 8ca97317-201e-00b3-50a7-08c411000000 blobfuse-rest.log-20220114"(1029739,557):Jan 13 18:01:55 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/common/area_log/log_lunar_prod_bcm_d_2022-01-11_20220114_010045.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=144947&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Thu, 13 Jan 2022 18:01:55 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 5639c8c4-401e-00ce-0ca7-0858d9000000 blobfuse-rest.log-20220114"(1029743,540):Jan 13 18:01:55 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/imexi/log_imexi_2022-01-12_P644_20220114_004739.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=81240&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Thu, 13 Jan 2022 18:01:55 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 904126b3-a01e-00a9-18a7-08eb7e000000 blobfuse-rest.log-20220114"(1217948,567):Jan 13 21:31:00 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/common/ci_cust_basic_excls/ci_excls_ssme_cust_seg_2022-01-12_20220114_043030.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=131&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Thu, 13 Jan 2022 21:31:00 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 8ce9d157-201e-00b3-79c4-08c411000000 blobfuse-rest.log-20220114"(1217949,547):Jan 13 21:31:00 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/scf_inc/log_scf_inc_2022-01-13_P27945_20220114_040153.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=225503&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Thu, 13 Jan 2022 21:31:00 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 74cce731-101e-0055-6ec4-08d5e1000000 blobfuse-rest.log-20220114"(1217950,539):Jan 13 21:31:00 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/win/log_win_2022-01-12_P3075_20220113_164654.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=3970542&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Thu, 13 Jan 2022 21:31:00 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 25280dac-e01e-008c-21c4-0873cd000000 blobfuse-rest.log-20220114"(1218340,560):Jan 13 21:31:11 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/common/area_log/log_ci_cust_basic_excls_2022-01-12_20220114_043021.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=168343&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Thu, 13 Jan 2022 21:31:11 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : a8e8973e-901e-00a1-3bc4-08f00d000000 blobfuse-rest.log-20220114"(1218342,539):Jan 13 21:31:11 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/win/log_win_2022-01-12_P3075_20220113_164654.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=3970542&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Thu, 13 Jan 2022 21:31:11 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 92b01954-901e-0126-50c4-08e377000000 blobfuse-rest.log-20220114"(1218343,539):Jan 13 21:31:11 ed11-hdi08p blobfuse[3964]: ==> REQUEST/RESPONSE :: PUT https://scbdipseasta005stdprd.blob.core.windows.net/framework-log/202201/raw/etl/win/log_win_2022-01-12_P3075_20220113_164654.log?sp=rwdl&st=2021-03-12T09:06:31Z&se=2099-12-31T09:06:00Z&sv=2020-02-10&sr=c&sig=REDACTED?&Content-Length=3970542&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.4.1&x-ms-date=Thu, 13 Jan 2022 21:31:11 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 400 :: REQ ID : 648a0e10-601e-00b4-68c4-083294000000

These logs are from Jan-13th. Server logs rolls over in 2 days, can you generate a new request which results in 400 and share the log.

Ok, then only I confirmed is below

From the URLs posted above its not the access key rather SAS that you are using for auth. And its always framework-log container hitting the response 400. Can you kindly check SAS mentioned in config file for that container is correct or not.

does this framework-log and framework-hdi40 containers belong to same account ?

Thanks vibhansa, Let me try to confirm with Cx above and get back to you with any updates.

I confirmed with cx, framework-log and framework-hdi40 containers belong to same Azure storage account. But still confirming storage auth.

Hi vibhansa, Good day!

Customer is using use the SAS Token for authentication to access the storage account, And confirmed they SAS mentioned in config file for these two storag container is correct. Otherwise, the mount path will not work. issue just happened sometime, not always.

Thanks

Can we filter the logs and validate for process 'blobfuse[3964]' no request has been successful not just PUT ? Initial analysis from server end says there is something wrong with the request. Have requested server team to go through logs and figure out what is missing or invalid.

Hi vibhansa

! /bin/bash

Post script for mount cloud_pool

Variables mount Azure disk.

Syntax sh edgenodescript_cloud_pool_prd.v0.1.sh ""

accoutname1=$1 accoutkeyob=$2

prepare directory for blobfuse

sudo mkdir -p /mnt/azureblobtmp/cloud_pool sudo mkdir -p /datagrid/cloud_pool

prepare blobfuse config file

sudo sh -c "echo 'accountName ${accoutname1}' >> /opt/blobfuse_config/blobfuse_cloud_pool.cfg" sudo sh -c "echo 'sasToken ${accoutkeyob}' >> /opt/blobfuse_config/blobfuse_cloud_pool.cfg" sudo sh -c "echo 'containerName cloud-pool' >> /opt/blobfuse_config/blobfuse_cloud_pool.cfg"

Change permission

sudo chmod 600 /opt/blobfuse_config/blobfuse_cloud_pool.cfg

use blobfuse to mount blob storage to file system

sudo blobfuse /datagrid/cloud_pool --tmp-path=/mnt/azureblobtmp/cloud_pool --config-file=/opt/blobfuse_config/blobfuse_cloud_pool.cfg --file-cache-timeout-in-seconds=0 -o attr_timeout=240 -o entry_timeout=240 -o negative_timeout=120 -o allow_other

Setting auto start

sudo sh -c "echo 'sudo blobfuse /datagrid/cloud_pool --tmp-path=/mnt/azureblobtmp/cloud_pool --config-file=/opt/blobfuse_config/blobfuse_cloud_pool.cfg --file-cache-timeout-in-seconds=0 -o attr_timeout=240 -o entry_timeout=240 -o negative_timeout=120 -o allow_other' >> /etc/init.d/mountblobfuse"

Hi vibhansa, Do our server team have some finding? Thanks.

Dint receive any reply yet.

I was looking at some more logs and iterating through the conversation again. I have one doubt here. Shell script that is shared in description is just checking whether a given path is mounted using blobfuse or not. Why there are overlaps in that path. If "/datagrid/cloud_pool"is mounted using blobfuse then none of its child directory shall be mounted again, because those child directory represent a directory on storage container and not meant to be mounted. Is my understanding on the overlap mount correct here ?

Kindly confirm it was overlapping mount points for blobfuse. In so its not supported and shall always be avoided. Each blobfuse instance needs a unique mount point not overlapping any other blobfuse instance otherwise behavior is undefined and may hit lot of issues. Try correcting the mount paths and validate things go back to normal post that.

Closing this as there is no update. As mentioned earlier on this thread, overlapping mounts are not supported. Even after resolving that if the issue persists feel free to reopen this thread.

Azure / azure-storage-fuse

Blobfuse that mount overlap often disconnect #680

Which version of the blobfuse was used?

Which OS (please include version) are you using?

What problem was encountered?

Shell Script:

for PT in $i do out=`ps -ef |grep blobfuse | grep -v grep | awk -F' ' '{print $9}' | grep "$PT$"` if [[ $PT != $out ]] then echo "$PT :: blob not found" fi done }

Ref

Have you found a mitigation/solution?

By default, blobfuse logs errors to syslog. If this is relevant, is there anything in the syslog that might be helpful?

If relevant, please share your mount command.

https://docs.microsoft.com/en-us/azure/hdinsight/domain-joined/hdinsight-security-overview

With this setup, enterprise employees can sign in to the cluster nodes by using their domain credentials. They can also use their domain credentials to authenticate with other approved endpoints. Like Apache Ambari Views, ODBC, JDBC, PowerShell, and REST APIs to interact with the cluster.

! /bin/bash

Post script for mount cloud_pool

Variables mount Azure disk.

Syntax sh edgenodescript_cloud_pool_prd.v0.1.sh ""

prepare directory for blobfuse

prepare blobfuse config file

Change permission

use blobfuse to mount blob storage to file system

Setting auto start

Azure / azure-storage-fuse

Blobfuse that mount overlap often disconnect #680

Which version of the blobfuse was used?

Which OS (please include version) are you using?

What problem was encountered?

Shell Script:

for PT in $i do out=ps -ef |grep blobfuse | grep -v grep | awk -F' ' '{print $9}' | grep "$PT$" if [[ $PT != $out ]] then echo "$PT :: blob not found" fi done }

Ref

Have you found a mitigation/solution?

By default, blobfuse logs errors to syslog. If this is relevant, is there anything in the syslog that might be helpful?

If relevant, please share your mount command.

https://docs.microsoft.com/en-us/azure/hdinsight/domain-joined/hdinsight-security-overview

With this setup, enterprise employees can sign in to the cluster nodes by using their domain credentials. They can also use their domain credentials to authenticate with other approved endpoints. Like Apache Ambari Views, ODBC, JDBC, PowerShell, and REST APIs to interact with the cluster.

! /bin/bash

Post script for mount cloud_pool

Variables mount Azure disk.

Syntax sh edgenodescript_cloud_pool_prd.v0.1.sh ""

prepare directory for blobfuse

prepare blobfuse config file

Change permission

use blobfuse to mount blob storage to file system

Setting auto start

for PT in $i do out=`ps -ef |grep blobfuse | grep -v grep | awk -F' ' '{print $9}' | grep "$PT$"` if [[ $PT != $out ]] then echo "$PT :: blob not found" fi done }