403 permission error when tpu-vm writing cloud bucket

[hazalturkmen]

Thank you for your research work. When I run the train.py script following the below code in tpu-vm, I got a 403 permission error.

python3 official/nlp/train.py --tpu=con-bert1 --experiment=bert/pretraining --mode=train --model_dir=gs://con_bioberturk/general/ --config_file=gs://con_bioberturk/bert_base.yaml --config_file=gs://con_bioberturk/pretrain.yaml --params_override="task.init_checkpoint=gs://con_bioberturk/bert-base-turkish-cased-tf/model.ckpt"

and my output is below:

`Saving experiment configuration to gs://con_bioberturk/general/params.yaml Traceback (most recent call last): File "/usr/share/tpu/models/official/modeling/hyperparams/params_dict.py", line 349, in save_params_dict_to_yaml yaml.dump(params.as_dict(), f, default_flow_style=False) File "/usr/local/lib/python3.8/dist-packages/yaml/init.py", line 290, in dump return dump_all([data], stream, Dumper=Dumper, **kwds) File "/usr/local/lib/python3.8/dist-packages/yaml/init.py", line 278, in dump_all dumper.represent(data) File "/usr/local/lib/python3.8/dist-packages/yaml/representer.py", line 28, in represent self.serialize(node) File "/usr/local/lib/python3.8/dist-packages/yaml/serializer.py", line 55, in serialize self.emit(DocumentEndEvent(explicit=self.use_explicit_end)) File "/usr/local/lib/python3.8/dist-packages/yaml/emitter.py", line 115, in emit self.state() File "/usr/local/lib/python3.8/dist-packages/yaml/emitter.py", line 220, in expect_document_end self.flush_stream() File "/usr/local/lib/python3.8/dist-packages/yaml/emitter.py", line 790, in flush_stream self.stream.flush() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/lib/io/file_io.py", line 219, in flush self._writable_file.flush() tensorflow.python.framework.errors_impl.PermissionDeniedError: Error executing an HTTP request: HTTP response code 403 with body '{ "error": { "code": 403, "message": "Access denied.", "errors": [ { "message": "Access denied.", "domain": "global", "reason": "forbidden" } ] } } ' when initiating an upload to gs://con_bioberturk/general/params.yaml

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "official/nlp/train.py", line 82, in app.run(main) File "/usr/local/lib/python3.8/dist-packages/absl/app.py", line 308, in run _run_main(main, args) File "/usr/local/lib/python3.8/dist-packages/absl/app.py", line 254, in _run_main sys.exit(main(argv)) File "official/nlp/train.py", line 47, in main train_utils.serialize_config(params, model_dir) File "/usr/share/tpu/models/official/core/train_utils.py", line 370, in serialize_config hyperparams.save_params_dict_to_yaml(params, params_save_path) File "/usr/share/tpu/models/official/modeling/hyperparams/params_dict.py", line 349, in save_params_dict_to_yaml yaml.dump(params.as_dict(), f, default_flow_style=False) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/lib/io/file_io.py", line 197, in exit self.close() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/lib/io/file_io.py", line 239, in close self._writable_file.close() tensorflow.python.framework.errors_impl.PermissionDeniedError: Error executing an HTTP request: HTTP response code 403 with body '{ "error": { "code": 403, "message": "Access denied.", "errors": [ { "message": "Access denied.", "domain": "global", "reason": "forbidden" } ] } } ' when initiating an upload to gs://con_bioberturk/general/params.yaml D1115 07:49:03.392797610 82369 init.cc:213] grpc_shutdown starts clean-up now`

Here is my settings:

• tpu-vm name:con-bert1
• TPU software version: tpu-vm-tf-2.10.0-pod
• cloud bucket (con_bioberturk) and tpu-vm are in the same location

[laxmareddyp]

Hi @hazalturkmen ,

This platform is focused on Bugs and Feature request, for support related questions, please use Tensorflow Forum and StackOverflow. For training and guides to use Tensorflow framework, please refer here Thanks!

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No