upload error when adding a job

[vanhauser-thc]

adding a local job results in an upload error:

| ERROR:root:Failed to get blob from key 58a7790f-e22f-4612-bc9d-62ad223f0b7d.
| Traceback (most recent call last):
|   File "/root/.local/share/virtualenvs/clusterfuzz-rAL0Uxhl/bin/gunicorn", line 8, in <module>
|     sys.exit(run())
|   File "/root/.local/share/virtualenvs/clusterfuzz-rAL0Uxhl/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 58, in run
|     WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
|   File "/root/.local/share/virtualenvs/clusterfuzz-rAL0Uxhl/lib/python3.8/site-packages/gunicorn/app/base.py", line 228, in run
|     super().run()
|   File "/root/.local/share/virtualenvs/clusterfuzz-rAL0Uxhl/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in run
|     Arbiter(self).run()
|   File "/root/.local/share/virtualenvs/clusterfuzz-rAL0Uxhl/lib/python3.8/site-packages/gunicorn/arbiter.py", line 202, in run
|     self.manage_workers()
|   File "/root/.local/share/virtualenvs/clusterfuzz-rAL0Uxhl/lib/python3.8/site-packages/gunicorn/arbiter.py", line 545, in manage_workers
|     self.spawn_workers()
|   File "/root/.local/share/virtualenvs/clusterfuzz-rAL0Uxhl/lib/python3.8/site-packages/gunicorn/arbiter.py", line 616, in spawn_workers
|     self.spawn_worker()
|   File "/root/.local/share/virtualenvs/clusterfuzz-rAL0Uxhl/lib/python3.8/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
|     worker.init_process()
|   File "/root/.local/share/virtualenvs/clusterfuzz-rAL0Uxhl/lib/python3.8/site-packages/gunicorn/workers/base.py", line 140, in init_process
|     self.run()
|   File "/root/.local/share/virtualenvs/clusterfuzz-rAL0Uxhl/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 123, in run
|     self.run_for_one(timeout)
|   File "/root/.local/share/virtualenvs/clusterfuzz-rAL0Uxhl/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 67, in run_for_one
|     self.accept(listener)
|   File "/root/.local/share/virtualenvs/clusterfuzz-rAL0Uxhl/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 29, in accept
|     self.handle(listener, client, addr)
|   File "/root/.local/share/virtualenvs/clusterfuzz-rAL0Uxhl/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 134, in handle
|     self.handle_request(listener, req, client, addr)
|   File "/root/.local/share/virtualenvs/clusterfuzz-rAL0Uxhl/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 175, in handle_request
|     respiter = self.wsgi(environ, resp.start_response)
|   File "third_party/flask/app.py", line 2464, in __call__
|     return self.wsgi_app(environ, start_response)
|   File "/clusterfuzz/src/appengine/server.py", line 96, in middleware
|     return wsgi_app(environ, start_response)
|   File "third_party/flask/app.py", line 2447, in wsgi_app
|     response = self.full_dispatch_request()
|   File "third_party/flask/app.py", line 1950, in full_dispatch_request
|     rv = self.dispatch_request()
|   File "third_party/flask/app.py", line 1936, in dispatch_request
|     return self.view_functions[rule.endpoint](**req.view_args)
|   File "third_party/flask/views.py", line 89, in view
|     return self.dispatch_request(*args, **kwargs)
|   File "/clusterfuzz/src/appengine/handlers/base_handler.py", line 282, in dispatch_request
|     return super().dispatch_request(*args, **kwargs)
|   File "/clusterfuzz/src/appengine/handlers/base_handler.py", line 271, in dispatch_request
|     return super(Handler, self).dispatch_request(*args, **kwargs)
|   File "third_party/flask/views.py", line 163, in dispatch_request
|     return meth(*args, **kwargs)
|   File "/clusterfuzz/src/appengine/libs/handler.py", line 369, in wrapper
|     response = make_response(func(self))
|   File "/clusterfuzz/src/appengine/libs/handler.py", line 292, in wrapper
|     return func(self, *args, **kwargs)
|   File "/clusterfuzz/src/appengine/libs/handler.py", line 438, in wrapper
|     return func(self, *args, **kwargs)
|   File "/clusterfuzz/src/appengine/handlers/jobs.py", line 177, in post
|     blob_info = self.get_upload()
|   File "/clusterfuzz/src/appengine/handlers/base_handler.py", line 293, in get_upload
|     blob_info = storage.GcsBlobInfo.from_key(upload_key)
|   File "/clusterfuzz/src/python/google_cloud_utils/storage.py", line 557, in from_key
|     logs.log_error('Failed to get blob from key %s.' % key)
| LogError: Failed to get blob from key 58a7790f-e22f-4612-bc9d-62ad223f0b7d.
| Traceback (most recent call last):
|   File "/clusterfuzz/src/python/google_cloud_utils/storage.py", line 555, in from_key
|     return GcsBlobInfo(blobs_bucket(), key)
|   File "/clusterfuzz/src/python/google_cloud_utils/storage.py", line 537, in __init__
|     self.filename = gcs_object['metadata'].get(BLOB_FILENAME_METADATA_KEY)
| TypeError: 'NoneType' object is not subscriptable
| ERROR:root:Failed to upload.
| Traceback (most recent call last):
|   File "/clusterfuzz/src/appengine/handlers/base_handler.py", line 271, in dispatch_request
|     return super(Handler, self).dispatch_request(*args, **kwargs)
|   File "third_party/flask/views.py", line 163, in dispatch_request
|     return meth(*args, **kwargs)
|   File "/clusterfuzz/src/appengine/libs/handler.py", line 369, in wrapper
|     response = make_response(func(self))
|   File "/clusterfuzz/src/appengine/libs/handler.py", line 292, in wrapper
|     return func(self, *args, **kwargs)
|   File "/clusterfuzz/src/appengine/libs/handler.py", line 438, in wrapper
|     return func(self, *args, **kwargs)
|   File "/clusterfuzz/src/appengine/handlers/jobs.py", line 177, in post
|     blob_info = self.get_upload()
|   File "/clusterfuzz/src/appengine/handlers/base_handler.py", line 295, in get_upload
|     raise helpers.EarlyExitException('Failed to upload.', 500)
| libs.helpers.EarlyExitException: Failed to upload.

[inferno-chromium]

Is go installed and go server running, see https://github.com/google/clusterfuzz/issues/1715#issuecomment-609570127 [go is used for run an emulated gcs locally - see code - https://github.com/google/clusterfuzz/blob/master/src/local/butler/run_server.py#L166]

[vanhauser-thc]

yes go is installed. Jonathan found a workaround by putting the file into a specific directory.

[vanhauser-thc]

I want to add that @jonathanmetzman encountered exactly the same issue on his system.

[inferno-chromium]

i cannot reproduce, so lets see :)

Two things to try:

1. i hope you ran run_server with --bootstrap. python butler.py run_server --bootstrap

2. try ctrl+c after run_server runs then do

   <from clusterfuzz checkout root dir>
   cd local
   go run emulators/gcs.go -storage-path=<path-to-clusterfuzz-root>/local/storage/local_gcs

   Does that exit with any error?

Next, how big is the archive (preferrably zip) that you are trying to add?

[vanhauser-thc]

1. i hope you ran run_server with --bootstrap.
   python butler.py run_server --bootstrap

sure did, and only the first time. restarted the server too several times

2. try ctrl+c after run_server runs
   then do

<from clusterfuzz checkout root dir>
cd local
go run emulators/gcs.go -storage-path=<path-to-clusterfuzz-root>/local/storage/local_gcs

no error, it just ... hangs there, does not return, no output.

Next, how big is the archive (preferrably zip) that you are trying to add?

very small, it was brotli. 10MB

[Vancir]

I made the following changes to support uploading files remotely to the local instance of ClusterFuzz.

1. replace the localhost in local/emulators/gcs.go with 0.0.0.0

   - http.ListenAndServe(fmt.Sprintf("localhost:%d", port), nil)
   + http.ListenAndServe(fmt.Sprintf("0.0.0.0:%d", port), nil)

Please turn off the firewall rules to make sure you can access the port. You can use the following command to verify whether you can access the port.

$ curl http://<your server ip address>:9008
ERROR: invalid path.

2. relace the localhost in src/local/butler/constants.py with your server ip address

   - LOCAL_GCS_SERVER_HOST = "http://localhost:" + str(LOCAL_GCS_SERVER_PORT)
   + LOCAL_GCS_SERVER_HOST = "http://" + "<your server ip address here>" + ":" + str(LOCAL_GCS_SERVER_PORT)

Of course, you can run the following python code to get your server ip address automatically.

import socket 
address = socket.gethostbyname(socket.gethostname())
print(address)