Error downloading files that were uploaded in the UI

[vchendrix]

Created a dataset in the ui, then uploaded a zip file to it. The error below was encountered when trying to download the file.

Internal Server Error: /api/v1/datasets/27/archive/

AttributeError at /api/v1/datasets/27/archive/ 'NoneType' object has no attribute 'replace'

Request Method: GET Request URL: https://ngt-dev.lbl.gov/api/v1/datasets/27/archive/ Django Version: 1.10.4 Python Executable: /usr/bin/uwsgi-core Python Version: 3.4.3 Python Path: ['.', '', '/var/www/django/ngt-archive/ngt-dev.lbl.gov/lib/python3.4', '/var/www/django/ngt-archive/ngt-dev.lbl.gov/lib/python3.4/plat-x86_64-linux-gnu', '/var/www/django/ngt-archive/ngt-dev.lbl.gov/lib/python3.4/lib-dynload', '/usr/lib/python3.4', '/usr/lib/python3.4/plat-x86_64-linux-gnu', '/var/www/django/ngt-archive/ngt-dev.lbl.gov/lib/python3.4/site-packages'] Server time: Fri, 16 Dec 2016 23:03:55 +0000 Installed Applications: ['ngt_archive', 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'archive_api.apps.ArchiveApiConfig', 'ui.apps.UiConfig', 'rest_framework'] Installed Middleware: ['django.middleware.security.SecurityMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.auth.middleware.SessionAuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware']

Traceback:

File "/var/www/django/ngt-archive/ngt-dev.lbl.gov/lib/python3.4/site-packages/django/core/handlers/exception.py" in inner

39. response = get_response(request)

File "/var/www/django/ngt-archive/ngt-dev.lbl.gov/lib/python3.4/site-packages/django/core/handlers/base.py" in _legacy_get_response

249. response = self._get_response(request)

File "/var/www/django/ngt-archive/ngt-dev.lbl.gov/lib/python3.4/site-packages/django/core/handlers/base.py" in _get_response

187. response = self.process_exception_by_middleware(e, request)

File "/var/www/django/ngt-archive/ngt-dev.lbl.gov/lib/python3.4/site-packages/django/core/handlers/base.py" in _get_response

185. response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/var/www/django/ngt-archive/ngt-dev.lbl.gov/lib/python3.4/site-packages/django/views/decorators/csrf.py" in wrapped_view

58. return view_func(*args, **kwargs)

File "/var/www/django/ngt-archive/ngt-dev.lbl.gov/lib/python3.4/site-packages/rest_framework/viewsets.py" in view

87. return self.dispatch(request, *args, **kwargs)

File "/var/www/django/ngt-archive/ngt-dev.lbl.gov/lib/python3.4/site-packages/rest_framework/views.py" in dispatch

466. response = self.handle_exception(exc)

File "/var/www/django/ngt-archive/ngt-dev.lbl.gov/lib/python3.4/site-packages/rest_framework/views.py" in dispatch

463. response = handler(request, *args, **kwargs)

File "/var/www/django/ngt-archive/ngt-dev.lbl.gov/lib/python3.4/site-packages/archive_api/viewsets.py" in archive

85. datasetname = dataset.name.replace(" ", "")

Exception Type: AttributeError at /api/v1/datasets/27/archive/ Exception Value: 'NoneType' object has no attribute 'replace' Request information: USER: valerie

GET: No GET data

POST: No POST data

FILES: No FILES data

COOKIES: sessionid = 'hvws2k4s5ec0xuhcigjqson43kqz6plj' _ga = 'GA1.2.925868728.1480364893' csrftoken = 'ZZ4xcjSMrSJjjISbz590VPuX0e9AOnkUziAbPA9TYzyIeWfmcoJNao9LXuWEaDqV' tabstyle = 'raw-tab'

META: CSRF_COOKIE = 'ZZ4xcjSMrSJjjISbz590VPuX0e9AOnkUziAbPA9TYzyIeWfmcoJNao9LXuWEaDqV' HTTP_ACCEPT = 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8' HTTP_ACCEPT_ENCODING = 'gzip, deflate, sdch, br' HTTP_ACCEPT_LANGUAGE = 'en-US,en;q=0.8' HTTP_CONNECTION = 'Keep-Alive' HTTP_COOKIE = '_ga=GA1.2.925868728.1480364893; tabstyle=raw-tab; sessionid=hvws2k4s5ec0xuhcigjqson43kqz6plj; csrftoken=ZZ4xcjSMrSJjjISbz590VPuX0e9AOnkUziAbPA9TYzyIeWfmcoJNao9LXuWEaDqV' HTTP_HOST = 'ngt-dev.lbl.gov' HTTP_REFERER = 'https://ngt-dev.lbl.gov/api/v1/datasets/27/' HTTP_UPGRADE_INSECURE_REQUESTS = '1' HTTP_USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36' HTTP_X_FORWARDED_FOR = '76.220.74.138' HTTP_X_FORWARDED_HOST = 'ngt-dev.lbl.gov' HTTP_X_FORWARDED_PROTO = 'https' HTTP_X_FORWARDED_SERVER = 'ngt-dev.lbl.gov' PATH_INFO = '/api/v1/datasets/27/archive/' QUERY_STRING = '' REMOTE_ADDR = '127.0.0.1' REQUEST_METHOD = 'GET' REQUEST_URI = '/api/v1/datasets/27/archive/' SCRIPT_NAME = '' SERVER_NAME = 'babe.lbl.gov' SERVER_PORT = '62000' SERVER_PROTOCOL = 'HTTP/1.1' uwsgi.node = b'babe.lbl.gov' uwsgi.version = b'1.9.17.1-debian' wsgi.errors = <_io.TextIOWrapper name=2 mode='w' encoding='ANSI_X3.4-1968'> wsgi.file_wrapper = '' wsgi.input = <uwsgi._Input object at 0x7fadc7e29600> wsgi.multiprocess = False wsgi.multithread = False wsgi.run_once = False wsgi.url_scheme = 'http' wsgi.version =

Settings: Using settings module ngt_archive.settings ABSOLUTE_URL_OVERRIDES = {} ADMINS = ALLOWED_HOSTS = ['*'] APPEND_SLASH = True AUTHENTICATION_BACKENDS = AUTH_LDAP_BIND_DN = 'CN=NGT-Archive,CN=Users,DC=flux,DC=local' AUTH_LDAP_BIND_PASSWORD = '****' AUTH_LDAP_CACHE_GROUPS = True AUTH_LDAP_CONNECTION_OPTIONS = {8: 0} AUTH_LDAP_GROUP_CACHE_TIMEOUT = 300 AUTH_LDAP_SERVER_URI = 'ldap://scooby.lbl.gov' AUTH_LDAP_USER_ATTR_MAP = {'last_name': 'sn', 'first_name': 'givenName', 'email': 'mail'} AUTH_LDAP_USER_SEARCH = <django_auth_ldap.config.LDAPSearch object at 0x7fadcb2557b8> AUTH_PASSWORD_VALIDATORS = '****' AUTH_USER_MODEL = 'auth.User' BASE_DIR = '/var/www/django/ngt-archive/ngt-dev.lbl.gov/lib/python3.4/site-packages' CACHES = {'default': {'BACKEND': 'django.core.cache.backends.locmem.LocMemCache'}} CACHE_MIDDLEWARE_ALIAS = 'default' CACHE_MIDDLEWARE_KEY_PREFIX = '****' CACHE_MIDDLEWARE_SECONDS = 600 CSRF_COOKIE_AGE = 31449600 CSRF_COOKIE_DOMAIN = None CSRF_COOKIE_HTTPONLY = False CSRF_COOKIE_NAME = 'csrftoken' CSRF_COOKIE_PATH = '/' CSRF_COOKIE_SECURE = False CSRF_FAILURE_VIEW = 'django.views.csrf.csrf_failure' CSRF_HEADER_NAME = 'HTTP_X_CSRFTOKEN' CSRF_TRUSTED_ORIGINS = [] DATABASES = {'default': {'PASSWORD': '****', 'NAME': 'ngt_dev', 'PORT': '', 'AUTOCOMMIT': True, 'OPTIONS': {}, 'TEST': {'COLLATION': None, 'NAME': None, 'CHARSET': None, 'MIRROR': None}, 'ENGINE': 'django.db.backends.postgresql_psycopg2', 'ATOMIC_REQUESTS': False, 'HOST': 'localhost', 'CONN_MAX_AGE': 0, 'TIME_ZONE': None, 'USER': 'ngeetuser'}} DATABASE_ROUTERS = [] DATASET_ARCHIVE_ROOT = '/data/NGEET/archives/ngt-dev.lbl.gov' DATASET_ARCHIVE_URL = '/archives/' DATA_UPLOAD_MAX_MEMORY_SIZE = 2621440 DATA_UPLOAD_MAX_NUMBER_FIELDS = 1000 DATETIME_FORMAT = 'N j, Y, P' DATETIME_INPUT_FORMATS = ['%Y-%m-%d %H:%M:%S', '%Y-%m-%d %H:%M:%S.%f', '%Y-%m-%d %H:%M', '%Y-%m-%d', '%m/%d/%Y %H:%M:%S', '%m/%d/%Y %H:%M:%S.%f', '%m/%d/%Y %H:%M', '%m/%d/%Y', '%m/%d/%y %H:%M:%S', '%m/%d/%y %H:%M:%S.%f', '%m/%d/%y %H:%M', '%m/%d/%y'] DATE_FORMAT = 'N j, Y' DATE_INPUT_FORMATS = ['%Y-%m-%d', '%m/%d/%Y', '%m/%d/%y', '%b %d %Y', '%b %d, %Y', '%d %b %Y', '%d %b, %Y', '%B %d %Y', '%B %d, %Y', '%d %B %Y', '%d %B, %Y'] DEBUG = False DEBUG_PROPAGATE_EXCEPTIONS = False DECIMAL_SEPARATOR = '.' DEFAULT_CHARSET = 'utf-8' DEFAULT_CONTENT_TYPE = 'text/html' DEFAULT_EXCEPTION_REPORTER_FILTER = 'django.views.debug.SafeExceptionReporterFilter' DEFAULT_FILE_STORAGE = 'django.core.files.storage.FileSystemStorage' DEFAULT_FROM_EMAIL = 'webmaster@localhost' DEFAULT_INDEX_TABLESPACE = '' DEFAULT_TABLESPACE = '' DISALLOWED_USER_AGENTS = [] EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend' EMAIL_HOST = 'localhost' EMAIL_HOST_PASSWORD = '****' EMAIL_HOST_USER = '' EMAIL_PORT = 25 EMAIL_SSL_CERTFILE = None EMAIL_SSL_KEYFILE = '****' EMAIL_SUBJECT_PREFIX = '[Django] ' EMAIL_TIMEOUT = None EMAIL_USE_SSL = False EMAIL_USE_TLS = False FILE_CHARSET = 'utf-8' FILE_UPLOAD_DIRECTORY_PERMISSIONS = None FILE_UPLOAD_HANDLERS = ['django.core.files.uploadhandler.MemoryFileUploadHandler', 'django.core.files.uploadhandler.TemporaryFileUploadHandler'] FILE_UPLOAD_MAX_MEMORY_SIZE = 2621440 FILE_UPLOAD_PERMISSIONS = None FILE_UPLOAD_TEMP_DIR = None FIRST_DAY_OF_WEEK = 0 FIXTURE_DIRS = [] FORCE_SCRIPT_NAME = None FORMAT_MODULE_PATH = None IGNORABLE_404_URLS = [] INSTALLED_APPS = ['ngt_archive', 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'archive_api.apps.ArchiveApiConfig', 'ui.apps.UiConfig', 'rest_framework'] INTERNAL_IPS = [] LANGUAGES = [('af', 'Afrikaans'), ('ar', 'Arabic'), ('ast', 'Asturian'), ('az', 'Azerbaijani'), ('bg', 'Bulgarian'), ('be', 'Belarusian'), ('bn', 'Bengali'), ('br', 'Breton'), ('bs', 'Bosnian'), ('ca', 'Catalan'), ('cs', 'Czech'), ('cy', 'Welsh'), ('da', 'Danish'), ('de', 'German'), ('dsb', 'Lower Sorbian'), ('el', 'Greek'), ('en', 'English'), ('en-au', 'Australian English'), ('en-gb', 'British English'), ('eo', 'Esperanto'), ('es', 'Spanish'), ('es-ar', 'Argentinian Spanish'), ('es-co', 'Colombian Spanish'), ('es-mx', 'Mexican Spanish'), ('es-ni', 'Nicaraguan Spanish'), ('es-ve', 'Venezuelan Spanish'), ('et', 'Estonian'), ('eu', 'Basque'), ('fa', 'Persian'), ('fi', 'Finnish'), ('fr', 'French'), ('fy', 'Frisian'), ('ga', 'Irish'), ('gd', 'Scottish Gaelic'), ('gl', 'Galician'), ('he', 'Hebrew'), ('hi', 'Hindi'), ('hr', 'Croatian'), ('hsb', 'Upper Sorbian'), ('hu', 'Hungarian'), ('ia', 'Interlingua'), ('id', 'Indonesian'), ('io', 'Ido'), ('is', 'Icelandic'), ('it', 'Italian'), ('ja', 'Japanese'), ('ka', 'Georgian'), ('kk', 'Kazakh'), ('km', 'Khmer'), ('kn', 'Kannada'), ('ko', 'Korean'), ('lb', 'Luxembourgish'), ('lt', 'Lithuanian'), ('lv', 'Latvian'), ('mk', 'Macedonian'), ('ml', 'Malayalam'), ('mn', 'Mongolian'), ('mr', 'Marathi'), ('my', 'Burmese'), ('nb', 'Norwegian Bokmål'), ('ne', 'Nepali'), ('nl', 'Dutch'), ('nn', 'Norwegian Nynorsk'), ('os', 'Ossetic'), ('pa', 'Punjabi'), ('pl', 'Polish'), ('pt', 'Portuguese'), ('pt-br', 'Brazilian Portuguese'), ('ro', 'Romanian'), ('ru', 'Russian'), ('sk', 'Slovak'), ('sl', 'Slovenian'), ('sq', 'Albanian'), ('sr', 'Serbian'), ('sr-latn', 'Serbian Latin'), ('sv', 'Swedish'), ('sw', 'Swahili'), ('ta', 'Tamil'), ('te', 'Telugu'), ('th', 'Thai'), ('tr', 'Turkish'), ('tt', 'Tatar'), ('udm', 'Udmurt'), ('uk', 'Ukrainian'), ('ur', 'Urdu'), ('vi', 'Vietnamese'), ('zh-hans', 'Simplified Chinese'), ('zh-hant', 'Traditional Chinese')] LANGUAGES_BIDI = ['he', 'ar', 'fa', 'ur'] LANGUAGE_CODE = 'en-us' LANGUAGE_COOKIE_AGE = None LANGUAGE_COOKIE_DOMAIN = None LANGUAGE_COOKIE_NAME = 'django_language' LANGUAGE_COOKIE_PATH = '/' LOCALE_PATHS = [] LOGGING = {} LOGGING_CONFIG = 'logging.config.dictConfig' LOGIN_REDIRECT_URL = '/accounts/profile/' LOGIN_URL = '/accounts/login/' LOGOUT_REDIRECT_URL = None MANAGERS = MEDIA_ROOT = '' MEDIA_URL = '' MESSAGE_STORAGE = 'django.contrib.messages.storage.fallback.FallbackStorage' MIDDLEWARE = None MIDDLEWARE_CLASSES = ['django.middleware.security.SecurityMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.auth.middleware.SessionAuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware'] MIGRATION_MODULES = {} MONTH_DAY_FORMAT = 'F j' NUMBER_GROUPING = 0 PASSWORD_HASHERS = '****' PASSWORD_RESET_TIMEOUT_DAYS = '****' PREPEND_WWW = False REST_FRAMEWORK = {'DEFAULT_PERMISSION_CLASSES': ('rest_framework.permissions.IsAuthenticated', 'rest_framework.permissions.DjangoModelPermissions'), 'DEFAULT_RENDERER_CLASSES': ('rest_framework.renderers.JSONRenderer', 'rest_framework.renderers.BrowsableAPIRenderer'), 'DEFAULT_AUTHENTICATION_CLASSES': ('rest_framework.authentication.SessionAuthentication',), 'DEFAULT_VERSION': 'v1', 'TEST_REQUEST_DEFAULT_FORMAT': 'json', 'DEFAULT_VERSIONING_CLASS': 'rest_framework.versioning.NamespaceVersioning'} ROOT_URLCONF = 'ngt_archive.urls' SECRET_KEY = '****' SECURE_BROWSER_XSS_FILTER = False SECURE_CONTENT_TYPE_NOSNIFF = False SECURE_HSTS_INCLUDE_SUBDOMAINS = False SECURE_HSTS_SECONDS = 0 SECURE_PROXY_SSL_HEADER = SECURE_REDIRECT_EXEMPT = [] SECURE_SSL_HOST = None SECURE_SSL_REDIRECT = False SERVER_EMAIL = 'root@localhost' SESSION_CACHE_ALIAS = 'default' SESSION_COOKIE_AGE = 1209600 SESSION_COOKIE_DOMAIN = None SESSION_COOKIE_HTTPONLY = True SESSION_COOKIE_NAME = 'sessionid' SESSION_COOKIE_PATH = '/' SESSION_COOKIE_SECURE = False SESSION_ENGINE = 'django.contrib.sessions.backends.db' SESSION_EXPIRE_AT_BROWSER_CLOSE = False SESSION_FILE_PATH = None SESSION_SAVE_EVERY_REQUEST = False SESSION_SERIALIZER = 'django.contrib.sessions.serializers.JSONSerializer' SETTINGS_MODULE = 'ngt_archive.settings' SHORT_DATETIME_FORMAT = 'm/d/Y P' SHORT_DATE_FORMAT = 'm/d/Y' SIGNING_BACKEND = 'django.core.signing.TimestampSigner' SILENCED_SYSTEM_CHECKS = [] STATICFILES_DIRS = STATICFILES_FINDERS = ['django.contrib.staticfiles.finders.FileSystemFinder', 'django.contrib.staticfiles.finders.AppDirectoriesFinder'] STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.StaticFilesStorage' STATIC_ROOT = '/var/www/django/ngt-archive/ngt-dev.lbl.gov/static' STATIC_URL = '/static/' TEMPLATES = [{'BACKEND': 'django.template.backends.django.DjangoTemplates', 'APP_DIRS': True, 'OPTIONS': {'context_processors': ['django.template.context_processors.debug', 'django.template.context_processors.request', 'django.contrib.auth.context_processors.auth', 'django.contrib.messages.context_processors.messages']}}] TEST_NON_SERIALIZED_APPS = [] TEST_RUNNER = 'django.test.runner.DiscoverRunner' THOUSAND_SEPARATOR = ',' TIME_FORMAT = 'P' TIME_INPUT_FORMATS = ['%H:%M:%S', '%H:%M:%S.%f', '%H:%M'] TIME_ZONE = 'UTC' USE_ETAGS = False USE_I18N = True USE_L10N = True USE_THOUSAND_SEPARATOR = False USE_TZ = True USE_X_FORWARDED_HOST = False USE_X_FORWARDED_PORT = False WSGI_APPLICATION = 'ngt_archive.wsgi.application' X_FRAME_OPTIONS = 'SAMEORIGIN' YEAR_MONTH_FORMAT = 'F Y'

[vchendrix]

I am assigning this to both of us because I don't know if it is a backend or front end issue.

[meghasandesh]

I think this is because I have't added the front end support for this. I'll see if this is the case.

[vchendrix]

OK. I will hold off until you investigate on your end. Let me know if I need to look into anything.

[meghasandesh]

I created the UI for enabling the downloads. But it looks like all we get is a blank file. Trying out the URL for the file directly in a browser window did not help either. Not really sure what is going on. Will try and find out more.

[vchendrix]

Any progress on this?

[meghasandesh]

I looked at the output of the GET request but it looks like the payload is zero bytes. I am seeing the same behavior in the API browser, so I am not sure what the UI has to do with this. Here's a sample output of the GET call (API browser):

Allow:GET, HEAD, OPTIONS Content-Disposition:attachment; filename=NGT0001_1.0_Data_Set_1.zip Content-Type:text/html; charset=utf-8 Date:Tue, 03 Jan 2017 17:11:23 GMT Server:WSGIServer/0.2 CPython/3.4.3rc1 Vary:Accept, Cookie X-Frame-Options:SAMEORIGIN X-Sendfile:/Users/MSandesh/ngt-archive-megha/archives/NGT0001_1.0.zip

[vchendrix]

does the file/Users/MSandesh/ngt-archive-megha/archives/NGT0001_1.0.zip exist on your desktop? Also, is it complete?

[meghasandesh]

Yes and yes

[vchendrix]

So, you are not getting the original error logged for this github issue?

Internal Server Error: /api/v1/datasets/27/archive/

AttributeError at /api/v1/datasets/27/archive/ 'NoneType' object has no attribute 'replace'

[meghasandesh]

Well, no. This is the command line output:

[03/Jan/2017 17:08:12] "GET /api/v1/datasets/1/ HTTP/1.1" 200 21914 [03/Jan/2017 17:08:17] "GET /api/v1/datasets/1/archive HTTP/1.1" 301 0 [03/Jan/2017 17:08:17] "GET /api/v1/datasets/1/archive/ HTTP/1.1" 200 0 [03/Jan/2017 17:11:23] "GET /api/v1/datasets/1/archive/ HTTP/1.1" 200 0 [03/Jan/2017 17:20:35] "GET /api/v1/ HTTP/1.1" 200 5538 [03/Jan/2017 17:20:47] "GET /api/v1/datasets/1/archive/ HTTP/1.1" 200 0

[vchendrix]

Originally the file was not being uploaded properly by the UI. Now it is being uploaded by the UI but not being downloaded. Is this correct?

[vchendrix]

Was file /Users/MSandesh/ngt-archive-megha/archives/NGT0001_1.0.zip uploaded through the UI?

[meghasandesh]

Yes, the file gets uploaded properly; i.e., shows up in the archive/ folder as a zip file - like NGT0001_1.0.zip. Note that this isn't the name of the zip file I uploaded. Unzipping it gives me the contents of the upload.

[vchendrix]

Your particular download issue is that XsendFile can only be properly handled by an apache server. Therefore, using the django webserver will not handle downloads correctly. I have just confirmed that download is not a problem in this case in my development apache server. The bug logged for this issue was that upload did not upload a file.

I will confirm that this is still the case.

[vchendrix]

I was able to reproduce the original error. However, the file was uploaded successfully. I will look into this further to see what more information I can glean. Seems like this is possibly a backend issue.