Azcopy failed even with SAS token in the VinVL_DOWNLOAD.md file

[Rosiekk]

          > Note: The data is on Azure Storage Blob, a SAS with Read permission is provided. Please append the following SAS at the end of each link to download:

?sp=r&st=2023-08-28T01:12:41Z&se=3023-08-28T09:12:41Z&sv=2022-11-02&sr=c&sig=6R1YmWluiXmPLsdVn1rDUpeBp2SYBMxDjc6KoKNlY8Q%3D

Thanks for the SAS token! However, when I appended the SAS token to download the pre-extracted features in the VinVL_DOWNLOAD.md file, I met the following issues: ./azcopy cp "https://biglmdiag.blob.core.windows.net/vinvl/image_features/oi_X152C4_frcnnbig2_exp168model_0060000model.roi_heads.nm_filter_2_model.roi_heads.score_thresh_0.2/model_0060000?sp=r&st=2023-08-28T01:12:41Z&se=3023-08-28T09:12:41Z&sv=2022-11-02&sr=c&sig=6R1YmWluiXmPLsdVn1rDUpeBp2SYBMxDjc6KoKNlY8Q%3D" [local_path] --recursive

INFO: Scanning...
INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support

failed to perform copy command due to error: cannot start job due to error: cannot list files due to reason -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /home/vsts/go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.15.0/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=AuthenticationFailed) =====
Description=403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature., Details: (none)
   HEAD https://biglmdiag.blob.core.windows.net/vinvl/image_features/oi_X152C4_frcnnbig2_exp168model_0060000model.roi_heads.nm_filter_2_model.roi_heads.score_thresh_0.2/model_0060000?se=3023-08-28t09%3A12%3A41z&sig=-REDACTED-&sp=r&sr=c&st=2023-08-28t01%3A12%3A41z&sv=2022-11-02&timeout=901
   User-Agent: [AzCopy/10.20.1 Azure-Storage/0.15 (go1.19.12; linux)]
   X-Ms-Client-Request-Id: [aa911033-29be-4949-4899-d28b11af8e85]
   X-Ms-Version: [2020-10-02]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
   Date: [Sat, 02 Sep 2023 08:18:05 GMT]
   Server: [Microsoft-HTTPAPI/2.0]
   X-Ms-Error-Code: [AuthenticationFailed]
   X-Ms-Request-Id: [6f4fb951-c01e-00f1-0475-dd6d29000000]

Any ideas? Thank you very much!

Originally posted by @Rosiekk in https://github.com/microsoft/Oscar/issues/205#issuecomment-1703763202

[xuehaoliu109]

I am facing the same problem. It seems that the token is only for Oscar DOWNLOAD.md, and the VinVL data and models need their own token.

[mesnico]

I am experiencing the same problem. Since some of my works rely on VinVL data, @ChunyuanLI would it be possible for you to generate a SAS token for the VinVL folder too? Thank you very much!

[meharbhatia]

@mesnico @xuehaoliu109 @Rosiekk were you able to access the updated SAS token? My work relies on VinVL data as well. Please let me know if you have a solution!

[Nuclear2]

@mesnico @xuehaoliu109 @Rosiekk were you able to access the updated SAS token? My work relies on VinVL data as well. Please let me know if you have a solution!

Also waiting for the updated SAS token. It seems that the vinvl models were public-accessible at first, but to some reason they are not now. (See #77, I wonder why)

[Aashish75]

@ChunyuanLI @pzzhang @xjli @eaidova Can some please provide the updated SAS token or an alternative way to download the VinVL checkpoints?

[pzzhang]

@Aashish75 The VinVL checkpoints can be found here (as updated in https://github.com/pzzhang/VinVL):

pretrained models at https://drive.google.com/file/d/1nvu8y4zZFbJqSqLdQClvMyzsbBeX-oaQ/view?usp=sharing

the associated labelmap at https://drive.google.com/file/d/1M1nPtMPHS1GXx5HvKMMDxOksxQgwX14_/view?usp=sharing

python tools/test_sg_net.py --config-file sgg_configs/vgattr/vinvl_x152c4.yaml TEST.IMS_PER_BATCH 2 MODEL.WEIGHT models/vinvl/vinvl_vg_x152c4.pth MODEL.ROI_HEADS.NMS_FILTER 1 MODEL.ROI_HEADS.SCORE_THRESH 0.2 DATA_DIR "../maskrcnn-benchmark-1/datasets1" TEST.IGNORE_BOX_REGRESSION True MODEL.ATTRIBUTE_ON True TEST.OUTPUT_FEATURE True

[Aashish75]

@pzzhang I am actually looking for the pre-trained Oscar+ model which is located in the "https://biglmdiag.blob.core.windows.net/vinvl/model_ckpts/TASK_NAME" path as per [https://github.com/microsoft/Oscar/blob/master/VinVL_DOWNLOAD.md]. To be more clear, I'm looking for vinvl/model_ckpts/vqa/base/checkpoint-2000000 and datasets/vqa/cache/trainval_ans2label.pkl to fine-tune/infer on VQA downstream task. I'm following the instructions mentioned in https://github.com/microsoft/Oscar/blob/master/VinVL_MODEL_ZOO.md to do this. The links you shared are referring to the pre-trained OD model instead. Please correct me if I'm wrong.

[pzzhang]

Oh, the checkpoint you are looking for is pre-trained VL transformer. What I shared is the VinVL visual feature extractor. They are different.

For the pre-trained VL transformer, I do not have access to it any more. @ChunyuanLI , do you still have access?