Problem with uploading competition bundles to Azure VM

[abmnv]

Hi, I installed an instance of codalab and codalab-cli on Microsoft Azure VM running Ubunty 14.10. I followed codalab documentation for setting up storage and service bus and modifying corresponding configuration files. The codalab web interface works and I can create user accounts and upload datasets. However, I still can't upload competition bundles, I am getting an error : "There was an error uploading the file. Please try again.". I tried uploading the same competition bundle to codalab.org and it worked. Please, let me know what the problem might be. thanks, artem

[ckcollab]

Artem,

Do you have any other error details other than "There was an error uploading the file."? Any logs from the server or anything? There could be many problems, maybe directory permissions?

Thanks!

[abmnv]

This is output from the server. It doesn't seem to contain any errors. What permissions should I check? My Azure storage has two containers: one is "bundles" which has access set to private and the other one is "public" with "public blob" access. thanks

2015-02-18 17:02:55,238 DEBUG _generate_blob_sas_url: sas=https://portalvhdsmqm8wtllyn34k.blob.core.windows.net/bundles/competition/upload/2/c9e010de-2848-4cb5-8ea4-30234130d009.zip?st=2015-02-18T16:57:55Z&se=2015-02-18T17:57:55Z&sp=w&sr=b&sig=S2U0Vjy/Bek2ZbnM4j1R/QmXTSrq5SPRtUgL3Y/7pDs%3D&; blob_name=competition/upload/2/c9e010de-2848-4cb5-8ea4-30234130d009.zip. DEBUG:apps.api.views.competition_views:_generate_blob_sas_url: sas=https://portalvhdsmqm8wtllyn34k.blob.core.windows.net/bundles/competition/upload/2/c9e010de-2848-4cb5-8ea4-30234130d009.zip?st=2015-02-18T16:57:55Z&se=2015-02-18T17:57:55Z&sp=w&sr=b&sig=S2U0Vjy/Bek2ZbnM4j1R/QmXTSrq5SPRtUgL3Y/7pDs%3D&; blob_name=competition/upload/2/c9e010de-2848-4cb5-8ea4-30234130d009.zip. [18/Feb/2015 17:02:55] "POST /api/competition/create/sas HTTP/1.1" 201 340

[ckcollab]

There is no exception given or any more debug details? That's strange... really hard to say. Could you get me any more details/logs/stack traces?

[abmnv]

How can I access these logs?

[ckcollab]

I'm not sure where yours are output, it depends on your server configuration. We'll need some kind of stack trace to figure out what's wrong.

[abmnv]

I have a basic development server configuration as described in codalab documentation. Could you give some hints how I can generate stack trace for this kind of problem? thanks.

[ckcollab]

Which documentation did you use? Look for any files *.log, like "webapp.log"

[abmnv]

these three were the most useful: https://github.com/codalab/codalab/blob/master/README.md, https://github.com/codalab/codalab/wiki/Dev_Set-up-data-storage, https://github.com/codalab/codalab/wiki/Dev_Configure-CodaLab-for-Development

[abmnv]

I don't have any *.log files in subdirectories of codalab or codalab-cli.

[ckcollab]

I would look into Django logging and try to get that working, once you have logging setup you should be able to get a stack trace at that point.

[abmnv]

The debug logging is on but it is not very helpful. The error is coming from this javaScript file /codalab/codalab/apps/web/static/codalab.js

[ckcollab]

What is the actual error?

[abmnv]

There was an error uploading the file. Please try again.

[ckcollab]

Sorry I can't be of more assistance, but I'll need a lot more information to diagnose the problem :(

Make sure all of your credentials are setup properly, that's the only thing I can think of.

[abmnv]

All settings seem to be fine. I set them up according to this document: https://github.com/codalab/codalab/blob/master/README.md, https://github.com/codalab/codalab/wiki/Dev_Set-up-data-storage. Do you remember any specific files/settings I should check?

[ckcollab]

How are you running the server? There might be a flag to specify where output should go...

Also, you could try adding a service like BugSnag to get the details of the error?

[abmnv]

I run it in the standard development mode ./runserver 0.0.0.0:8000. Right now all output goes to stdout but I can change that. I will take a look into BugSnag.

[ckcollab]

The stdout also isn't displaying useful stack traces or anything?

[abmnv]

Nope, just this: 2015-02-18 17:02:55,238 DEBUG _generate_blob_sas_url: sas=https://portalvhdsmqm8wtllyn34k.blob.core.windows.net/bundles/competition/upload/2/c9e010de-2848-4cb5-8ea4-30234130d009.zip?st=2015-02-18T16:57:55Z&se=2015-02-18T17:57:55Z&sp=w&sr=b&sig=S2U0Vjy/Bek2ZbnM4j1R/QmXTSrq5SPRtUgL3Y/7pDs%3D&; blob_name=competition/upload/2/c9e010de-2848-4cb5-8ea4-30234130d009.zip. DEBUG:apps.api.views.competition_views:_generate_blob_sas_url: sas=https://portalvhdsmqm8wtllyn34k.blob.core.windows.net/bundles/competition/upload/2/c9e010de-2848-4cb5-8ea4-30234130d009.zip?st=2015-02-18T16:57:55Z&se=2015-02-18T17:57:55Z&sp=w&sr=b&sig=S2U0Vjy/Bek2ZbnM4j1R/QmXTSrq5SPRtUgL3Y/7pDs%3D&; blob_name=competition/upload/2/c9e010de-2848-4cb5-8ea4-30234130d009.zip. [18/Feb/2015 17:02:55] "POST /api/competition/create/sas HTTP/1.1" 201 340

[abmnv]

Is there a working Azure VM for testing where I can check settings and test competition submission?

[ckcollab]

Try out http://codalabtest.cloudapp.net

[abmnv]

Can I get user access to this machine to see how different settings are implemented?

[ckcollab]

No unfortunately I don't think that will be possible. Could you take a screenshot of the page with the error message?

[abmnv]

[ckcollab]

Yeah that looks like something is definitely going wrong with permissions, have you tried stepping through the code/adding prints to relevant places?

Checkout apps/api/views/competition_views.py and see if you can pin-point where it may be getting stuck?

[abmnv]

I will try that.

[ckcollab]

It might be helpful to screen share on skype, feel free to add/bug me: the_924 is my name on sykpe!

[abmnv]

ok, thanks.

[abmnv]

Thanks Eric for your help, I solved this problem with CORS. I found this script inside codalab package that worked: codalab/codalabtools/compute/enableCORS.py

[ckcollab]

Yesssssssss! Glad it works for ya!

[abmnv]

I am getting another error though now. Here is the stack trace. Looks like a problem with Service Bus? Thanks.

2015-02-27 20:26:33,934 DEBUG _generate_blob_sas_url: sas=https://portalvhdsmqm8wtllyn34k.blob.core.windows.net/bundles/competition/upload/2/666c5b57-522b-4892-9c6e-827b34110c88.zip?st=2015-02-27T20:21:33Z&se=2015-02-27T21:21:33Z&sp=w&sr=b&sig=hHo5u97CgQ8iBJ6b2qMNXE46IZ2JzfsB9158JQid93c%3D&; blob_name=competition/upload/2/666c5b57-522b-4892-9c6e-827b34110c88.zip. 2015-02-27 20:26:33,935 DEBUG response data: {'url': 'https://portalvhdsmqm8wtllyn34k.blob.core.windows.net/bundles/competition/upload/2/666c5b57-522b-4892-9c6e-827b34110c88.zip?st=2015-02-27T20:21:33Z&se=2015-02-27T21:21:33Z&sp=w&sr=b&sig=hHo5u97CgQ8iBJ6b2qMNXE46IZ2JzfsB9158JQid93c%3D&', 'version': '2013-08-15', 'id': 'competition/upload/2/666c5b57-522b-4892-9c6e-827b34110c88.zip'} [27/Feb/2015 20:26:33] "POST /api/competition/create/sas HTTP/1.1" 201 340 [27/Feb/2015 20:26:33] "GET /static/fonts/segoe-ui_semibold.woff HTTP/1.1" 304 0 2015-02-27 20:26:34,349 DEBUG CompetitionCreation: owner=2; filename=competition/upload/2/666c5b57-522b-4892-9c6e-827b34110c88.zip. [27/Feb/2015 20:26:34] "GET /static/img/PreloaderTransp.gif HTTP/1.1" 200 3213 2015-02-27 20:26:34,467 DEBUG CompetitionCreation def: owner=2; def=2; blob=competition/upload/2/666c5b57-522b-4892-9c6e-827b34110c88.zip. 2015-02-27 20:26:34,535 ERROR Internal Server Error: /api/competition/create Traceback (most recent call last): File "/home/azureuser/codalab/venv/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 113, in get_response response = callback(request, _callback_args, _callback_kwargs) File "/home/azureuser/codalab/venv/local/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view return self.dispatch(request, _args, _kwargs) File "/home/azureuser/codalab/venv/local/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 77, in wrapped_view return view_func(_args, _kwargs) File "/home/azureuser/codalab/venv/local/lib/python2.7/site-packages/rest_framework/views.py", line 327, in dispatch response = self.handle_exception(exc) File "/home/azureuser/codalab/venv/local/lib/python2.7/site-packages/rest_framework/views.py", line 324, in dispatch response = handler(request, _args, _kwargs) File "/home/azureuser/codalab/codalab/apps/api/views/competition_views.py", line 98, in post job = create_competition(cdb.pk) File "/home/azureuser/codalab/codalab/apps/web/tasks.py", line 103, in create_competition return Job.objects.create_and_dispatch_job('create_competition', {'comp_def_id': comp_def_id}) File "/home/azureuser/codalab/codalab/apps/jobs/models.py", line 72, in create_and_dispatch_job getQueue(queue_name).send_message(job.create_json_message()) File "/home/azureuser/codalab/codalab/codalabtools/azure_extensions.py", line 81, in send_message self._try_request(op, fail=fail) File "/home/azureuser/codalab/codalab/codalabtools/azure_extensions.py", line 57, in _try_request return fn() File "/home/azureuser/codalab/codalab/codalabtools/azure_extensions.py", line 79, in op = lambda: self.service.send_queue_message(self.name, Message(body)) File "/home/azureuser/codalab/venv/local/lib/python2.7/site-packages/azure/servicebus/servicebusservice.py", line 642, in send_queue_message request.headers = self._update_service_bus_header(request) File "/home/azureuser/codalab/venv/local/lib/python2.7/site-packages/azure/servicebus/servicebusservice.py", line 803, in _update_service_bus_header request.headers.append(('Authorization', self._sign_service_bus_request(request))) File "/home/azureuser/codalab/venv/local/lib/python2.7/site-packages/azure/servicebus/servicebusservice.py", line 810, in _sign_service_bus_request return 'WRAP access_token="' + self._get_token(request.host, request.path) + '"' File "/home/azureuser/codalab/venv/local/lib/python2.7/site-packages/azure/servicebus/servicebusservice.py", line 847, in _get_token resp = self._httpclient.perform_request(request) File "/home/azureuser/codalab/venv/local/lib/python2.7/site-packages/azure/http/httpclient.py", line 144, in perform_request self.send_request_headers(connection, request.headers) File "/home/azureuser/codalab/venv/local/lib/python2.7/site-packages/azure/http/httpclient.py", line 125, in send_request_headers connection.endheaders() File "/usr/lib/python2.7/httplib.py", line 997, in endheaders self._send_output(message_body) File "/usr/lib/python2.7/httplib.py", line 850, in _send_output self.send(msg) File "/usr/lib/python2.7/httplib.py", line 812, in send self.connect() File "/usr/lib/python2.7/httplib.py", line 1200, in connect self.timeout, self.source_address) File "/usr/lib/python2.7/socket.py", line 553, in create_connection for res in getaddrinfo(host, port, 0, SOCK_STREAM): gaierror: [Errno -2] Name or service not known [27/Feb/2015 20:26:34] "POST /api/competition/create HTTP/1.1" 500 19092

[ckcollab]

Not sure about this error, I think you're right though something is not set right to do with service bus stuff.

Double triple check that .codalabconfig and local.py, maybe add a couple prints somewhere around here: File "/home/azureuser/codalab/codalab/codalabtools/azure_extensions.py", line 81, in send_message

That might give you some clues as to what the bad value might be?

[ckcollab]

Is this issue resolved now?

[abmnv]

Hi Eric. It looks like the problem is that my Service Bus doesn't have ACS authentication, only SAS. Basically, codalab can't connect to accesscontrol.windows.net domain, only servicebus.windows.net. As I understand these days when a Service Bus is manually created in the Azure portal only SAS is activated and not ACS. Although It used to be both but recently changed. Now the only way to activate ACS is through PoweShell script. Does this problem sound familiar? I will try using PowerShell today.

[abmnv]

ok, I solved the problem. I created a new Service Bus account with ACS enabled in PowerShell using this command and it worked: New-AzureSBNamespace -Name mynamespace -Location "East US" -CreateACSNamespace $true -NamespaceType Messaging