Legacy Import - Performance Problem

[missyschoenbaum]

for RC5.2 build Steps to recreate open app select Open Select Import NAADSM Legacy Scenario Use the Choose File button to select a parameter file Use the Choose File button to select a population file: have image - Image may insert at the bottom

Select the Save changes button

get big yellow errror ValueError at /app/ImportScenario/ The view ADSMSettings.views.import_naadsm_scenario didn't return an HttpResponse object. It returned None instead. Request Method: POST Request URL: http://127.0.0.1:8000/app/ImportScenario/ Django Version: 1.7.1 Exception Type: ValueError Exception Value:
The view ADSMSettings.views.import_naadsm_scenario didn't return an HttpResponse object. It returned None instead. Exception Location: C:\Users\meschoenbaum\Desktop\ADSM_4_10\Lib\site-packages\django\core\handlers\base.py in get_response, line 130 Python Executable: C:\Users\meschoenbaum\Desktop\ADSM_4_10\adsm.exe Python Version: 3.4.2 Python Path:
['C:\Users\meschoenbaum\Desktop\ADSM_4_10\adsm.exe', 'C:\Users\meschoenbaum\Desktop\ADSM_4_10', 'C:\Users\meschoenbaum\Desktop\ADSM_4_10\adsm.zip', 'C:\Users\meschoenbaum\Desktop\ADSM_4_10\library.zip', 'C:\Users\meschoenbaum\Desktop\ADSM_4_10\src', 'C:\Users\meschoenbaum\Desktop\ADSM_4_10', 'C:\Users\meschoenbaum\Desktop\ADSM_4_10\DLLs', 'C:\Users\meschoenbaum\Desktop\ADSM_4_10\Lib', 'C:\Users\meschoenbaum\Desktop\ADSM_4_10\Lib\plat-win', 'C:\Users\meschoenbaum\Desktop\ADSM_4_10\Lib\lib-tk', 'C:\Users\meschoenbaum\Desktop\ADSM_4_10\Lib\site-packages', 'C:\Users\meschoenbaum\Desktop\ADSM_4_10\bin', 'C:\Users\meschoenbaum\Desktop\ADSM_4_10\Scripts', 'C:\Users\meschoenbaum\Desktop\ADSM_4_10\Lib\site-packages\cx_Freeze-4.3.4-py3.4-win-amd64.egg', 'C:\Users\meschoenbaum\Desktop\ADSM_4_10\Lib\site-packages\numpy-1.9.1-py3.4-win-amd64.egg', 'C:\Users\meschoenbaum\Desktop\ADSM_4_10\Lib\site-packages\pip-1.5.6-py3.4.egg', 'C:\Users\meschoenbaum\Desktop\ADSM_4_10\Lib\site-packages\pywin32-219-py3.4-win-amd64.egg', 'C:\Users\meschoenbaum\Desktop\ADSM_4_10\src'] Server time: Mon, 13 Apr 2015 16:05:56 +0000

Environment:

Request Method: POST Request URL: http://127.0.0.1:8000/app/ImportScenario/

Django Version: 1.7.1 Python Version: 3.4.2 Installed Applications: ('ScenarioCreator', 'Results', 'ADSMSettings', 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.humanize', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'floppyforms', 'crispy_forms', 'productionserver') Installed Middleware: ('django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'ADSMSettings.disable.DisableCSRF')

Traceback: File "C:\Users\meschoenbaum\Desktop\ADSM_4_10\Lib\site-packages\django\core\handlers\base.py" in get_response

1. % (callback.module, view_name))

Exception Type: ValueError at /app/ImportScenario/ Exception Value: The view ADSMSettings.views.import_naadsm_scenario didn't return an HttpResponse object. It returned None instead. population_xml
<InMemoryUploadedFile: oldnaadsmpop.xml (text/xml)> parameters_xml
<InMemoryUploadedFile: oldnaadsm.xml (text/xml)>

wsgi.multiprocess
False wsgi.input
<cherrypy.wsgiserver.wsgiserver3.KnownLengthRFile object at 0x00000000095C1E48> HTTPACCEPT 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/_;q=0.8' wsgi.errors

<_io.TextIOWrapper name='' mode='w' encoding='cp437'> QUERY_STRING '' HTTP_CONNECTION 'close' HTTP_HOST '127.0.0.1:8000' PATH_INFO '/app/ImportScenario/' REMOTE_PORT '62608' HTTP_REFERER 'http://127.0.0.1:8000/app/ImportScenario/' wsgi.multithread True HTTP_ORIGIN 'http://127.0.0.1:8000' REQUEST_METHOD 'POST' CONTENT_TYPE 'multipart/form-data; boundary=----WebKitFormBoundaryiD7qfxkBhEkNAqkk' wsgi.url_scheme 'http' SERVER_NAME '127.0.0.1' HTTP_X_REAL_IP '127.0.0.1' REQUEST_URI '/app/ImportScenario/' wsgi.run_once False HTTP_ACCEPT_ENCODING 'gzip,deflate' CONTENT_LENGTH '2076757' HTTP_ACCEPT_LANGUAGE 'en-us,en;q=0.8' wsgi.version (1, 0) HTTP_CACHE_CONTROL 'max-age=0' HTTP_X_FORWARDED_FOR '127.0.0.1' ACTUAL_SERVER_PROTOCOL 'HTTP/1.1' SERVER_SOFTWARE 'CherryPy/3.6.0 Server' SERVER_PROTOCOL 'HTTP/1.0' REMOTE_ADDR '127.0.0.1' HTTP_USER_AGENT 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like ' 'Gecko) Chrome/37.0.2062.94 Safari/537.36' SCRIPT_NAME '' SERVER_PORT '8001' LOCALE_PATHS () STATICFILES_FINDERS ('django.contrib.staticfiles.finders.FileSystemFinder', 'django.contrib.staticfiles.finders.AppDirectoriesFinder') MEDIA_URL '/media/' DEFAULT_FROM_EMAIL 'webmaster@localhost' DEBUG_PROPAGATE_EXCEPTIONS False MONTH_DAY_FORMAT 'F j' WSGI_APPLICATION 'ADSM.wsgi.application' MEDIA_ROOT 'C:\Users\meschoenbaum\Desktop\ADSM_4_10\src\media' AUTHENTICATION_BACKENDS ('django.contrib.auth.backends.ModelBackend',) X_FRAME_OPTIONS 'SAMEORIGIN' SERVER_EMAIL 'root@localhost' USE_X_FORWARDED_HOST False DATETIME_FORMAT 'N j, Y, P' SEND_BROKEN_LINK_EMAILS False STATICFILES_STORAGE 'django.contrib.staticfiles.storage.StaticFilesStorage' FILE_UPLOAD_MAX_MEMORY_SIZE 2621440 CSRF_COOKIE_SECURE False EMAIL_USE_SSL False TEST_NON_SERIALIZED_APPS [] EMAIL_PORT 25 APPEND_SLASH True EMAIL_BACKEND 'django.core.mail.backends.smtp.EmailBackend' LOGGING_CONFIG 'logging.config.dictConfig' SESSION_SERIALIZER 'django.contrib.sessions.serializers.JSONSerializer' ALLOWED_HOSTS [] SESSION_ENGINE 'django.contrib.sessions.backends.db' SHORT_DATETIME_FORMAT 'm/d/Y P' TEMPLATE_DIRS ('C:\Users\meschoenbaum\Desktop\ADSM_4_10\src\templates',) DATETIME_INPUT_FORMATS ('%Y-%m-%d %H:%M:%S', '%Y-%m-%d %H:%M:%S.%f', '%Y-%m-%d %H:%M', '%Y-%m-%d', '%m/%d/%Y %H:%M:%S', '%m/%d/%Y %H:%M:%S.%f', '%m/%d/%Y %H:%M', '%m/%d/%Y', '%m/%d/%y %H:%M:%S', '%m/%d/%y %H:%M:%S.%f', '%m/%d/%y %H:%M', '%m/%d/%y') CSRF_COOKIE_NAME 'csrftoken' CSIDL_PERSONAL 5 SESSION_COOKIE_PATH '/' SESSION_FILE_PATH None SECRET_KEY '*******************_' LOGIN_URL '/accounts/login/' MESSAGE_STORAGE 'django.contrib.messages.storage.fallback.FallbackStorage' TRANSACTIONS_MANAGED False YEAR_MONTH_FORMAT 'F Y' BASE_DIR 'C:\Users\meschoenbaum\Desktop\ADSM_4_10\src' SESSION_COOKIE_HTTPONLY True DATE_INPUT_FORMATS ('%Y-%m-%d', '%m/%d/%Y', '%m/%d/%y', '%b %d %Y', '%b %d, %Y', '%d %b %Y', '%d %b, %Y', '%B %d %Y', '%B %d, %Y', '%d %B %Y', '%d %B, %Y') CACHES {'default': {'BACKEND': 'django.core.cache.backends.locmem.LocMemCache'}} TEMPLATE_STRING_IF_INVALID '' SESSION_COOKIE_SECURE False CSRF_COOKIE_AGE 31449600 USE_TZ True EMAIL_HOST_USER '' SHGFP_TYPE_CURRENT 0 USE_L10N True FILE_UPLOAD_PERMISSIONS None MIDDLEWARE_CLASSES ('django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'ADSMSettings.disable.DisableCSRF') DECIMAL_SEPARATOR '.' EMAIL_SUBJECT_PREFIX '[Django] ' EMAIL_USE_TLS False STATIC_ROOT 'C:\Users\meschoenbaum\Desktop\ADSM_4_10\src\static' TIME_ZONE 'UTC' ABSOLUTE_URL_OVERRIDES {} STATIC_URL '/static/' FORMAT_MODULE_PATH None FILE_CHARSET 'utf-8' INTERNAL_IPS () TIME_FORMAT 'P' TEMPLATE_LOADERS ('django.template.loaders.filesystem.Loader', 'django.template.loaders.app_directories.Loader') LANGUAGE_COOKIE_NAME 'django_language' LANGUAGES_BIDI ('he', 'ar', 'fa', 'ur') LOGGING {} ADMINS () DISALLOWED_USER_AGENTS () COMMENTS_ALLOW_PROFANITIES False DEFAULT_FILE_STORAGE 'django.core.files.storage.FileSystemStorage' DEFAULT_CHARSET 'utf-8' TEST_RUNNER 'django.test.runner.DiscoverRunner' USE_THOUSAND_SEPARATOR False TIME_INPUT_FORMATS ('%H:%M:%S', '%H:%M:%S.%f', '%H:%M') LANGUAGE_COOKIE_AGE None LANGUAGE_COOKIE_PATH '/' PROFANITIES_LIST '**_****************_' TEMPLATE_DEBUG True FILE_UPLOAD_TEMP_DIR None DB_BASE_DIR 'C:\Users\meschoenbaum\Documents\ADSM Workspace\settings' ADMIN_FOR () CACHE_MIDDLEWARE_ALIAS 'default' DATABASES {'default': {'ATOMIC_REQUESTS': False, 'AUTOCOMMIT': True, 'CONN_MAX_AGE': 0, 'ENGINE': 'django.db.backends.sqlite3', 'HOST': '', 'NAME': 'C:\Users\meschoenbaum\Documents\ADSM ' 'Workspace\settings\settings.sqlite3', 'OPTIONS': {}, 'PASSWORD': '**_****************_', 'PORT': '', 'TEST': {'CHARSET': None, 'COLLATION': None, 'MIRROR': None, 'NAME': 'C:\Users\meschoenbaum\Documents\ADSM ' 'Workspace\settings\test_settings.sqlite3'}, 'TIME_ZONE': 'UTC', 'USER': ''}, 'scenario_db': {'ATOMIC_REQUESTS': False, 'AUTOCOMMIT': True, 'CONN_MAX_AGE': 0, 'ENGINE': 'django.db.backends.sqlite3', 'HOST': '', 'NAME': 'C:\Users\meschoenbaum\Documents\ADSM ' 'Workspace\settings\activeSession.sqlite3', 'OPTIONS': {'timeout': 300}, 'PASSWORD': '**_****************_', 'PORT': '', 'TEST': {'CHARSET': None, 'COLLATION': None, 'MIRROR': None, 'NAME': 'C:\Users\meschoenbaum\Documents\ADSM ' 'Workspace\settings\test_activeSession.sqlite3'}, 'TIME_ZONE': 'UTC', 'USER': ''}} TEMPLATE_CONTEXT_PROCESSORS ('django.contrib.auth.context_processors.auth', 'django.core.context_processors.debug', 'django.core.context_processors.i18n', 'django.core.context_processors.media', 'django.core.context_processors.static', 'django.core.context_processors.tz', 'django.contrib.messages.context_processors.messages', 'django.core.context_processors.request', 'ADSMSettings.context_processor.adsm_context', 'ScenarioCreator.context_processor.basic_context', 'Results.context_processor.results_context') INSTALLED_APPS ('ScenarioCreator', 'Results', 'ADSMSettings', 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.humanize', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'floppyforms', 'crispy_forms', 'productionserver') CSRF_FAILURE_VIEW 'django.views.csrf.csrf_failure' DEBUG True CSRF_COOKIE_HTTPONLY False ALLOWED_INCLUDE_ROOTS () DEFAULT_EXCEPTION_REPORTER_FILTER 'django.views.debug.SafeExceptionReporterFilter' LANGUAGE_COOKIE_DOMAIN None DATABASE_ROUTERS ['ScenarioCreator.router.ScenarioRouter'] CSRF_COOKIE_DOMAIN None SETTINGS_MODULE 'ADSM.settings' DEFAULT_INDEX_TABLESPACE '' FILE_UPLOAD_HANDLERS ('django.core.files.uploadhandler.MemoryFileUploadHandler', 'django.core.files.uploadhandler.TemporaryFileUploadHandler') USE_ETAGS False SIGNING_BACKEND 'django.core.signing.TimestampSigner' SESSION_COOKIE_AGE 1209600 EMAIL_HOST 'localhost' CACHE_MIDDLEWARE_KEY_PREFIX '**_****************_' CSRF_COOKIE_PATH '/' SESSION_CACHE_ALIAS 'default' FIXTURE_DIRS () DATE_FORMAT 'N j, Y' SECURE_PROXY_SSL_HEADER None SESSION_SAVE_EVERY_REQUEST False SESSION_COOKIE_DOMAIN None PREPEND_WWW False MANAGERS () DEFAULT_TABLESPACE '' AUTH_USER_MODEL 'auth.User' SILENCED_SYSTEM_CHECKS [] FIRST_DAY_OF_WEEK 0 MIGRATION_MODULES {} CACHE_MIDDLEWARE_SECONDS 600 LOGIN_REDIRECT_URL '/accounts/profile/' PASSWORD_HASHERS '**_****************_' DEFAULT_CONTENT_TYPE 'text/html' STATICFILES_DIRS ('C:\Users\meschoenbaum\Desktop\ADSM_4_10\src\ADSM\static',) NUMBER_GROUPING 0 FORCE_SCRIPT_NAME None ROOT_URLCONF 'ADSM.urls' EMAIL_HOST_PASSWORD '**_****************_' FILE_UPLOAD_DIRECTORY_PERMISSIONS None LANGUAGE_CODE 'en-us' THOUSAND_SEPARATOR ',' PASSWORD_RESET_TIMEOUT_DAYS '**_*****************' USE_I18N True LANGUAGES (('af', 'Afrikaans'), ('ar', 'Arabic'), ('ast', 'Asturian'), ('az', 'Azerbaijani'), ('bg', 'Bulgarian'), ('be', 'Belarusian'), ('bn', 'Bengali'), ('br', 'Breton'), ('bs', 'Bosnian'), ('ca', 'Catalan'), ('cs', 'Czech'), ('cy', 'Welsh'), ('da', 'Danish'), ('de', 'German'), ('el', 'Greek'), ('en', 'English'), ('en-au', 'Australian English'), ('en-gb', 'British English'), ('eo', 'Esperanto'), ('es', 'Spanish'), ('es-ar', 'Argentinian Spanish'), ('es-mx', 'Mexican Spanish'), ('es-ni', 'Nicaraguan Spanish'), ('es-ve', 'Venezuelan Spanish'), ('et', 'Estonian'), ('eu', 'Basque'), ('fa', 'Persian'), ('fi', 'Finnish'), ('fr', 'French'), ('fy', 'Frisian'), ('ga', 'Irish'), ('gl', 'Galician'), ('he', 'Hebrew'), ('hi', 'Hindi'), ('hr', 'Croatian'), ('hu', 'Hungarian'), ('ia', 'Interlingua'), ('id', 'Indonesian'), ('io', 'Ido'), ('is', 'Icelandic'), ('it', 'Italian'), ('ja', 'Japanese'), ('ka', 'Georgian'), ('kk', 'Kazakh'), ('km', 'Khmer'), ('kn', 'Kannada'), ('ko', 'Korean'), ('lb', 'Luxembourgish'), ('lt', 'Lithuanian'), ('lv', 'Latvian'), ('mk', 'Macedonian'), ('ml', 'Malayalam'), ('mn', 'Mongolian'), ('mr', 'Marathi'), ('my', 'Burmese'), ('nb', 'Norwegian Bokmal'), ('ne', 'Nepali'), ('nl', 'Dutch'), ('nn', 'Norwegian Nynorsk'), ('os', 'Ossetic'), ('pa', 'Punjabi'), ('pl', 'Polish'), ('pt', 'Portuguese'), ('pt-br', 'Brazilian Portuguese'), ('ro', 'Romanian'), ('ru', 'Russian'), ('sk', 'Slovak'), ('sl', 'Slovenian'), ('sq', 'Albanian'), ('sr', 'Serbian'), ('sr-latn', 'Serbian Latin'), ('sv', 'Swedish'), ('sw', 'Swahili'), ('ta', 'Tamil'), ('te', 'Telugu'), ('th', 'Thai'), ('tr', 'Turkish'), ('tt', 'Tatar'), ('udm', 'Udmurt'), ('uk', 'Ukrainian'), ('ur', 'Urdu'), ('vi', 'Vietnamese'), ('zh-cn', 'Simplified Chinese'), ('zh-hans', 'Simplified Chinese'), ('zh-hant', 'Traditional Chinese'), ('zh-tw', 'Traditional Chinese')) SESSION_COOKIE_NAME 'sessionid' SHORT_DATE_FORMAT 'm/d/Y' SESSION_EXPIRE_AT_BROWSER_CLOSE False LOGOUT_URL '/accounts/logout/' CRISPY_TEMPLATE_PACK 'bootstrap' IGNORABLE_404_URLS ()

[missyschoenbaum]

I am having a legacy save issue. I could be that I am failing to get a status message, as I was using Texas and it is large.

Steps to recreate - Created new scenario when to Project panel, selected Import NAADSM Legacy Scenario Page to upload appeared I selected my parameter and pop files I hit apply, which then becomes inactive

So, it is not clear what my status is here - I could see it would take a long time to load Texas, but don't have any clue what is happening.

[missyschoenbaum]

Another possibility for weirdness here - I am using xml straight from an SC run, not an export from NAADSM. I don't know if they would be different.

[josiahseaman]

Definitely worth looking into in the next sprint. I'd say there's a 90% chance it's working but there's no progress bar implemented and the import is so huge it could take > 30 minutes. If that's the case then it can be fixed with progress bar and better xml2sqlite performance, which we've been talking about.

[missyschoenbaum]

I did pull up an empty scenario and load Texas, because I wanted to see how labels looked. It loaded in 31 minutes or less. I wasn't watching it closely, so I didn't capture the exact time.

Also, just the fact that it loads Texas and doesn't take down the app is an accomplishment!

[josiahseaman]

In my mind, there's a big difference between importing a population from within an existing blank scenario and importing NAADSM XML (which requires population). Are you saying you were able to pull in all your old parameters? If so, that's an impressive stress test, even if it is slow.

[missyschoenbaum]

No, I didn't get it to work on the legacy import screen. I just made a new scenario and pulled in the population only. I asked Michelle about her legacy import, and she was successful so I think we have more of a progress bar problem than an import problem.

[josiahseaman]

This is definitely a good place for a progress bar. Though progress bars are one of those things that are WAAAAY harder than you would think they should be: http://xkcd.com/612/ http://xkcd.com/1425/

[josiahseaman]

@ndh2 Hi Neil, this seems to be the closest issue to our hangouts discussions on XML2SQLite performance. We did some profiling on the import process and it is indeed dog slow for non-trivial cases.

sqlite "commit" method takes up over 85% of the total time. That means that waiting and performing on database modifications is essentially the entire problem. I have a fully parameterized example from Shaun with 2,727 units in it. Unit import (using bulk create already) takes a couple seconds. Parameter import takes upwards of 30 minutes on the same file. Essentially, bulk create needs to be used to build the parameter models in batch starting around xml2sqlite.py:400.

There's good news. I think that Relational and Probability Functions account for the bulk of all parameter content. If you can write a bulk create that sweeps through Functions and creates all of them first, that covers most all of your dependencies. Then you can go back through and probably build the rest of the models normally. At least Functions will be the biggest bang for your buck in terms of development time.

Related: #459 and #576

[sarahjgarza]

@ndh2 I seem to be getting the same error message every time I try to Import NAADSM Legacy Scenario. This is in Beta 3.3.5.13. Any ideas what could be wrong?

[ndh2]

@josiahseaman or @tjmahlin? The above looks to me like a file missing from a Bootstrap theme.

[ndh2]

Going to be posting some notes on bulk create to this thread. (No replies needed — just me thinking out loud.)

The script makes heavy use of create_no_duplicates, which calls get_or_create, which does an automatic save. So I will probably need to write a no-save version of get_or_create.

Question: does the "get" part of "get_or_create" examine in-memory objects (created but not yet saved to DB)?

[josiahseaman]

Answer: get_or_create() is a database call that doesn't know about unsaved objects or unsaved changes. I suppose you could make your own warehouse of in-memory objects to check.

[ndh2]

Commit dbdde3c provides a decent speedup for scenarios that are heavy on histogram or piecewise type PDFs.

[missyschoenbaum]

Missy needs to test with Texas again. @lkholmstrom can test with a scenario heavy on histogram or piecewise type pdfs. This may be solved, as I have Texas up and running at the moment.

[missyschoenbaum]

I have successfully loaded Texas again from scratch. I say we call this one good for now.