Sharing parameter blocks

[missyschoenbaum]

Josiah and I had discussions long ago about how we might have a method to share parameter blocks. For example, a way to export all disease parameters, email them to someone, and have the recipient easily upload them into a scenario.

This issue is to hold the question as a research item.

I would see this happening as part of the Functions panel. Currently the Rel functions can upload a data file, so we have one example of something importing. I would see the export file going into the output folder.

[missyschoenbaum]

Can we research the options on this?

[ConradSelig]

Relational functions would be easy, we could export one or all (or have both those options). PDFs would be difficult as there is currently no standard for file format. We would have to create a way to import/export - likely we would try to get this to match the relational formatting as closely as possible. Easiest option might actually be to export / import all functions instead of trying to work with individual functions. Some validation would have to be done on import to avoid duplicate function names - but that wouldn't be too difficult.

[missyschoenbaum]

On PDF's, it is just one table. It would seem we could just dump all the fields whether we needed them or not. Could we not use the same file format as the rel, just a text file? It would be hard for them to create from scratch, but I am not planning on having them create from scratch.

On Rel's, you have 2 tables, one with name, and one with points. I think we would retain the name they gave them. Or, were you thinking they would just use the import feature that they have now? I can write a query to extract just one rel if we want that to happen.

Also, on both types I am OK with saying you export all functions, and that you keep their names. It is some work to rename and delete, but less work than rebuilding.

[ConradSelig]

We should be able to dump all fields to .txt, like I said I dont know exactly how it's formatted but I can't see any reason why that wouldn't work. I would probably do auto-cleaning on name imports, if I find a duplicate I would just append " - imported" to the name. These changes shouldn't be too difficult. I would estimate about 6 hours including time to implement the buttons.

[ConradSelig]

I did a little work on this one and noticed you said you might want to be able to export Disease parameters as well as functions. How interested are you in exporting something like that? If we wanted to do disease blocks we would likely also want to do Controls. How would you want to split those up?

Here is a little sneak peek at what the function export is going to look like :)

[ConradSelig]

@missyschoenbaum do you have a .txt relational function? I would love to match that output with this export.

[ConradSelig]

Nevermind, I realized that mass export would have different formatting anyways. If you wanted to have individual function export then that is something I would need - I believe that is a different ticket however.

[missyschoenbaum]

Yes, i still owe you a comment on how to break up parameters.

[ConradSelig]

No rush, I'm currently working on the "Import Relational Function" and "Import Probability Density Functions" buttons.

[ConradSelig]

@missyschoenbaum Do you like how this looks? I formatted it so it also fits nicely on a laptop sized screen.

[missyschoenbaum]

Like it. Do you intend to leave text, or is that for me to make documentation?

[ConradSelig]

I changed the import line to have "current scenario's workspace" instead of "current workspace" to stay consistent with the export line. If you like the wording we can keep it, otherwise it would be very easy to change it right now if you want to take a stab at it.

[ConradSelig]

I moved the buttons and text inside the scroll container. This also prevents the buttons from showing when a function is opened. I re-formatted the buttons to fit on even smaller view-ports. Here is what it looks like now, note that I am scrolled all the way down:

[missyschoenbaum]

So the action I would take is to export, then copy into another scenario where I wished to import and as long as the file has REL or PDF? Also, notice one button is not plural, so add s. I like it.

[ConradSelig]

Action is correct. I think it makes sense to avoid files that contain the current scenario's name to avoid having a lot of duplicate functions (ie "RELSampleScenario.csv" would not import if you had "Sample Scenario" open). Does that action make sense? I changed import text to "Imported functions are read from csv files in the current scenario's workspace folder that have either "REL" or "PDF_" prepended to their names and do not contain the current scenario's name."

All files with delimiters will be imported (ie if you have multiple files with "REL_" in the name I'll import all functions from all files). Does the current wording correctly imply that?

Here is a sample of each function type export: sample exports.zip

[ConradSelig]

I changed the field order for exported pdfs to make importing easier. Here is a new sample file. PDF_SampleScenariowithOutputs.zip

[ConradSelig]

@missyschoenbaum Do you want any output to the terminal to indicate that new functions are being created? Also, is above action correct?

[missyschoenbaum]

I am thinking it should go fast. An insert and commit on this amount of data should be small. I guess the main thing is that they shouldn't be banging on it until it's done.

I think that action would be correct as best as I can guess right now. I opened the sample file. It would be perfect because you could remove things that you didn't want very easily by deleting the line.

[ConradSelig]

It does not take too long but I put a loading screen anyways because otherwise the user could stack instances of importing and that would cause issues. I won't output anything to the terminal.

[ConradSelig]

Functionally this ticket is complete - I'm moving to testing. I'm going to write all the comments for the new code and push that next.

[missyschoenbaum]

Just to confirm, I test the push out part and then the pull in part on another scenario.

[ConradSelig]

Assuming you are in a scenario called "Test910" when you export you would get either "REL_Test910.csv" or "PDFTest910.csv". These files will NOT import into Test910. These files can be moved into another scenario with an entirely different name OR you can rename the files. When I did my testing I renamed them to "[REL/PDF_]AllDuplicatesTest.csv", this new file will import and append " - imported" to all the function names.

" - imported" only appends to function names that are duplicate names (ie if there is not already a function with the same name the imported function name will not be changed).

Also if you already have a function called "Prevalence" and a function called "Prevalence - imported" and you import a function called "Prevalence" the program will append " - imported" until it is not a duplicate name so the final function name will be "Prevalence - imported - imported". The function will continue to append " - imported" until the name is not a duplicate regardless of how many times it has to append it.

[missyschoenbaum]

Rel Import worked great. PDF, not so much.

Will post full error text

[missyschoenbaum]

wsgi.version	(1, 0)
HTTP_ACCEPT	'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8'
HTTP_CONNECTION	'close'
SERVER_NAME	'127.0.0.1'
HTTP_REFERER	'http://127.0.0.1:63887/LoadingScreen/?loading_url=/setup/ImportFunctions/pdf'
REMOTE_ADDR	'127.0.0.1'
HTTP_ACCEPT_LANGUAGE	'en-us,en;q=0.8'
wsgi.run_once	False
wsgi.url_scheme	'http'
REQUEST_METHOD	'GET'
PATH_INFO	'/setup/ImportFunctions/pdf'
REQUEST_URI	'/setup/ImportFunctions/pdf'
QUERY_STRING	''
HTTP_X_FORWARDED_FOR	'127.0.0.1'
SERVER_PROTOCOL	'HTTP/1.0'
HTTP_USER_AGENT	'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like ' 'Gecko) Chrome/39.0.2171.95 Safari/537.36'
wsgi.errors	<_io.TextIOWrapper name='' mode='w' encoding='cp437'>
wsgi.multiprocess	False
wsgi.multithread	True
HTTP_HOST	'127.0.0.1:63887'
SERVER_PORT	'63888'
SERVER_SOFTWARE	'CherryPy/3.6.0 Server'
HTTP_ACCEPT_ENCODING	'gzip, deflate'
wsgi.input	<cherrypy.wsgiserver.wsgiserver3.KnownLengthRFile object at 0x000000000DD10550>
HTTP_X_REAL_IP	'127.0.0.1'
SCRIPT_NAME	''
REMOTE_PORT	'64085'
ACTUAL_SERVER_PROTOCOL	'HTTP/1.1'

Settings

ALLOWED_HOSTS	[]
TEMPLATE_DIRS	()
SETTINGS_MODULE	'ADSM.settings'
DEFAULT_CONTENT_TYPE	'text/html'
DATETIME_FORMAT	'N j, Y, P'
CACHE_MIDDLEWARE_ALIAS	'default'
SHGFP_TYPE_CURRENT	0
SHORT_DATE_FORMAT	'm/d/Y'
CSRF_COOKIE_HTTPONLY	False
AUTHENTICATION_BACKENDS	('django.contrib.auth.backends.ModelBackend',)
SECURE_HSTS_INCLUDE_SUBDOMAINS	False
NUMBER_GROUPING	0
OS_DIR	'windows'
SESSION_CACHE_ALIAS	'default'
DEFAULT_FILE_STORAGE	'django.core.files.storage.FileSystemStorage'
DEBUG_TOOLBAR_PANELS	[]
ABSOLUTE_URL_OVERRIDES	{}
DEFAULT_EXCEPTION_REPORTER_FILTER	'django.views.debug.SafeExceptionReporterFilter'
SECRET_KEY	'****'
LANGUAGES_BIDI	('he', 'ar', 'fa', 'ur')
BASE_DIR	'C:\Users\MeSchoenbaum\Desktop\ADSM_9_16'
SESSION_EXPIRE_AT_BROWSER_CLOSE	False
DEBUG_TOOLBAR_PATCH_SETTINGS	False
LANGUAGE_COOKIE_PATH	'/'
DATE_FORMAT	'N j, Y'
SILENCED_SYSTEM_CHECKS	[]
DEFAULT_INDEX_TABLESPACE	''
DISALLOWED_USER_AGENTS	()
DATE_INPUT_FORMATS	('%Y-%m-%d', '%m/%d/%Y', '%m/%d/%y', '%b %d %Y', '%b %d, %Y', '%d %b %Y', '%d %b, %Y', '%B %d %Y', '%B %d, %Y', '%d %B %Y', '%d %B, %Y')
TEST_RUNNER	'django.test.runner.DiscoverRunner'
EMAIL_HOST_PASSWORD	'****'
SESSION_COOKIE_SECURE	False
THOUSAND_SEPARATOR	','
TEMPLATE_LOADERS	('django.template.loaders.filesystem.Loader', 'django.template.loaders.app_directories.Loader')
SECURE_SSL_HOST	None
SERVER_EMAIL	'server@development.server'
EMAIL_USE_TLS	False
PASSWORD_RESET_TIMEOUT_DAYS	'****'
SESSION_SAVE_EVERY_REQUEST	False
DEBUG_TOOLBAR_CONFIG	{'JQUERY_URL': '', 'SHOW_TOOLBAR_CALLBACK': 'ddt_request_history.panels.request_history.allow_ajax'}
INSTALLED_APPS	('ScenarioCreator', 'Results', 'ADSMSettings', 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.humanize', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'floppyforms', 'crispy_forms', 'productionserver', 'webpack_loader', 'explorer')
FILE_UPLOAD_HANDLERS	('django.core.files.uploadhandler.MemoryFileUploadHandler', 'django.core.files.uploadhandler.TemporaryFileUploadHandler')
YEAR_MONTH_FORMAT	'F Y'
PRODUCTION_SETTINGS	False
EMAIL_SSL_KEYFILE	'****'
DEFAULT_FROM_EMAIL	'noreply@development.server'
CSRF_COOKIE_PATH	'/'
MIDDLEWARE_CLASSES	('django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'ADSMSettings.disable.DisableCSRF')
DEBUG	True
LANGUAGE_COOKIE_AGE	None
CACHES	{'default': {'BACKEND': 'django.core.cache.backends.locmem.LocMemCache'}}
LOGIN_REDIRECT_URL	'/'
ALLOWED_INCLUDE_ROOTS	()
EMAIL_BACKEND	'django.core.mail.backends.console.EmailBackend'
FILE_UPLOAD_PERMISSIONS	None
DEBUG_PROPAGATE_EXCEPTIONS	False
FIRST_DAY_OF_WEEK	0
PASSWORD_HASHERS	'****'
CSRF_COOKIE_SECURE	False
ADMINS	()
SECURE_SSL_REDIRECT	False
STATIC_ROOT	'C:\Users\MeSchoenbaum\Desktop\ADSM_9_16\static'
MESSAGE_STORAGE	'django.contrib.messages.storage.fallback.FallbackStorage'
FILE_CHARSET	'utf-8'
FILE_UPLOAD_TEMP_DIR	None
SECURE_PROXY_SSL_HEADER	None
STATICFILES_FINDERS	('django.contrib.staticfiles.finders.FileSystemFinder', 'django.contrib.staticfiles.finders.AppDirectoriesFinder')
LANGUAGE_CODE	'en-us'
EMAIL_TIMEOUT	None
INTERNAL_IPS	('127.0.0.1', '::1')
SESSION_COOKIE_NAME	'sessionid'
TEMPLATE_CONTEXT_PROCESSORS	('django.contrib.auth.context_processors.auth', 'django.template.context_processors.debug', 'django.template.context_processors.i18n', 'django.template.context_processors.media', 'django.template.context_processors.static', 'django.template.context_processors.tz', 'django.contrib.messages.context_processors.messages')
WSGI_APPLICATION	'ADSM.wsgi.application'
LOGIN_URL	'/accounts/login/'
SESSION_COOKIE_AGE	1209600
EMAIL_HOST_USER	''
DATABASE_ROUTERS	['ScenarioCreator.router.ScenarioRouter']
CSIDL_PERSONAL	5
LANGUAGE_COOKIE_DOMAIN	None
SESSION_SERIALIZER	'django.contrib.sessions.serializers.JSONSerializer'
EXPLORER_SQL_BLACKLIST	('ALTER', 'RENAME', 'DROP', 'TRUNCATE')
EXPLORER_DEFAULT_CONNECTION	'scenario_db'
SHORT_DATETIME_FORMAT	'm/d/Y P'
SCRIPT	'.cmd'
TEMPLATES	[{'APP_DIRS': True, 'BACKEND': 'django.template.backends.django.DjangoTemplates', 'DIRS': ['C:\Users\MeSchoenbaum\Desktop\ADSM_9_16\ADSM\templates', 'C:\Users\MeSchoenbaum\Desktop\ADSM_9_16\templates\ScenarioCreator', 'C:\Users\MeSchoenbaum\Desktop\ADSM_9_16\templates\Results', 'C:\Users\MeSchoenbaum\Desktop\ADSM_9_16\templates\ADSMSettings', 'C:\Users\MeSchoenbaum\Desktop\ADSM_9_16\templates\django.contrib.admin', 'C:\Users\MeSchoenbaum\Desktop\ADSM_9_16\templates\django.contrib.auth', 'C:\Users\MeSchoenbaum\Desktop\ADSM_9_16\templates\floppyforms', 'C:\Users\MeSchoenbaum\Desktop\ADSM_9_16\templates\crispy_forms', 'C:\Users\MeSchoenbaum\Desktop\ADSM_9_16\templates\explorer', 'C:\Users\MeSchoenbaum\Desktop\ADSM_9_16\templates\ADSM'], 'OPTIONS': {'context_processors': ['django.contrib.auth.context_processors.auth', 'django.core.context_processors.debug', 'django.core.context_processors.i18n', 'django.core.context_processors.media', 'django.core.context_processors.static', 'django.core.context_processors.tz', 'django.contrib.messages.context_processors.messages', 'django.core.context_processors.request', 'ADSMSettings.context_processor.adsm_context', 'ScenarioCreator.context_processor.basic_context', 'Results.context_processor.results_context']}}]
APPEND_SLASH	True
MEDIA_URL	'/media/'
PREPEND_WWW	False
SECURE_HSTS_SECONDS	0
MANAGERS	()
FILE_UPLOAD_DIRECTORY_PERMISSIONS	None
SECURE_CONTENT_TYPE_NOSNIFF	False
EMAIL_HOST	'localhost'
EXPLORER_SCHEMA_EXCLUDE_TABLE_PREFIXES	('auth', 'django', 'south_migrationhistory', 'sqlite_master', 'sqlite_sequence')
USE_X_FORWARDED_HOST	False
CSRF_COOKIE_DOMAIN	None
LOCALE_PATHS	()
CACHE_MIDDLEWARE_SECONDS	600
FIXTURE_DIRS	()
FILE_UPLOAD_MAX_MEMORY_SIZE	2621440
TEMPLATE_DEBUG	True
DB_BASE_DIR	'C:\Users\MeSchoenbaum\Documents\ADSM Beta Workspace\settings'
TEST_NON_SERIALIZED_APPS	[]
STATICFILES_DIRS	('C:\Users\MeSchoenbaum\Desktop\ADSM_9_16\ADSM\static',)
SESSION_COOKIE_HTTPONLY	True
USE_THOUSAND_SEPARATOR	False
STATIC_URL	'/static/'
ROOT_URLCONF	'ADSM.urls'
DECIMAL_SEPARATOR	'.'
EXTENSION	'.exe'
SECURE_BROWSER_XSS_FILTER	False
LANGUAGE_COOKIE_NAME	'django_language'
AUTH_USER_MODEL	'auth.User'
SIGNING_BACKEND	'django.core.signing.TimestampSigner'
TIME_ZONE	'UTC'
DEFAULT_TABLESPACE	''
SESSION_ENGINE	'django.contrib.sessions.backends.db'
TIME_INPUT_FORMATS	('%H:%M:%S', '%H:%M:%S.%f', '%H:%M')
WORKSPACE_PATH	'C:\Users\MeSchoenbaum\Documents\ADSM Beta Workspace'
DATETIME_INPUT_FORMATS	('%Y-%m-%d %H:%M:%S', '%Y-%m-%d %H:%M:%S.%f', '%Y-%m-%d %H:%M', '%Y-%m-%d', '%m/%d/%Y %H:%M:%S', '%m/%d/%Y %H:%M:%S.%f', '%m/%d/%Y %H:%M', '%m/%d/%Y', '%m/%d/%y %H:%M:%S', '%m/%d/%y %H:%M:%S.%f', '%m/%d/%y %H:%M', '%m/%d/%y')
LANGUAGES	(('af', 'Afrikaans'), ('ar', 'Arabic'), ('ast', 'Asturian'), ('az', 'Azerbaijani'), ('bg', 'Bulgarian'), ('be', 'Belarusian'), ('bn', 'Bengali'), ('br', 'Breton'), ('bs', 'Bosnian'), ('ca', 'Catalan'), ('cs', 'Czech'), ('cy', 'Welsh'), ('da', 'Danish'), ('de', 'German'), ('el', 'Greek'), ('en', 'English'), ('en-au', 'Australian English'), ('en-gb', 'British English'), ('eo', 'Esperanto'), ('es', 'Spanish'), ('es-ar', 'Argentinian Spanish'), ('es-mx', 'Mexican Spanish'), ('es-ni', 'Nicaraguan Spanish'), ('es-ve', 'Venezuelan Spanish'), ('et', 'Estonian'), ('eu', 'Basque'), ('fa', 'Persian'), ('fi', 'Finnish'), ('fr', 'French'), ('fy', 'Frisian'), ('ga', 'Irish'), ('gl', 'Galician'), ('he', 'Hebrew'), ('hi', 'Hindi'), ('hr', 'Croatian'), ('hu', 'Hungarian'), ('ia', 'Interlingua'), ('id', 'Indonesian'), ('io', 'Ido'), ('is', 'Icelandic'), ('it', 'Italian'), ('ja', 'Japanese'), ('ka', 'Georgian'), ('kk', 'Kazakh'), ('km', 'Khmer'), ('kn', 'Kannada'), ('ko', 'Korean'), ('lb', 'Luxembourgish'), ('lt', 'Lithuanian'), ('lv', 'Latvian'), ('mk', 'Macedonian'), ('ml', 'Malayalam'), ('mn', 'Mongolian'), ('mr', 'Marathi'), ('my', 'Burmese'), ('nb', 'Norwegian Bokmal'), ('ne', 'Nepali'), ('nl', 'Dutch'), ('nn', 'Norwegian Nynorsk'), ('os', 'Ossetic'), ('pa', 'Punjabi'), ('pl', 'Polish'), ('pt', 'Portuguese'), ('pt-br', 'Brazilian Portuguese'), ('ro', 'Romanian'), ('ru', 'Russian'), ('sk', 'Slovak'), ('sl', 'Slovenian'), ('sq', 'Albanian'), ('sr', 'Serbian'), ('sr-latn', 'Serbian Latin'), ('sv', 'Swedish'), ('sw', 'Swahili'), ('ta', 'Tamil'), ('te', 'Telugu'), ('th', 'Thai'), ('tr', 'Turkish'), ('tt', 'Tatar'), ('udm', 'Udmurt'), ('uk', 'Ukrainian'), ('ur', 'Urdu'), ('vi', 'Vietnamese'), ('zh-cn', 'Simplified Chinese'), ('zh-hans', 'Simplified Chinese'), ('zh-hant', 'Traditional Chinese'), ('zh-tw', 'Traditional Chinese'))
DATABASES	{'default': {'ATOMIC_REQUESTS': False, 'AUTOCOMMIT': True, 'CONN_MAX_AGE': 0, 'ENGINE': 'django.db.backends.sqlite3', 'HOST': '', 'NAME': 'C:\Users\MeSchoenbaum\Documents\ADSM Beta ' 'Workspace\settings\settings.db', 'OPTIONS': {}, 'PASSWORD': '****', 'PORT': '', 'TEST': {'CHARSET': None, 'COLLATION': None, 'MIRROR': None, 'NAME': 'C:\Users\MeSchoenbaum\Documents\ADSM ' 'Beta Workspace\settings\test_settings.db'}, 'TIME_ZONE': 'UTC', 'USER': ''}, 'scenario_db': {'ATOMIC_REQUESTS': False, 'AUTOCOMMIT': True, 'CONN_MAX_AGE': 0, 'ENGINE': 'django.db.backends.sqlite3', 'HOST': '', 'NAME': 'C:\Users\MeSchoenbaum\Documents\ADSM Beta ' 'Workspace\settings\activeSession.db', 'OPTIONS': {'timeout': 300}, 'PASSWORD': '****', 'PORT': '', 'TEST': {'CHARSET': None, 'COLLATION': None, 'MIRROR': None, 'NAME': 'C:\Users\MeSchoenbaum\Documents\ADSM ' 'Beta ' 'Workspace\settings\test_activeSession.db'}, 'TIME_ZONE': 'UTC', 'USER': ''}}
TEMPLATE_STRING_IF_INVALID	''
MIGRATION_MODULES	{}
CACHE_MIDDLEWARE_KEY_PREFIX	'****'
LOGGING_CONFIG	'logging.config.dictConfig'
USE_ETAGS	False
CSRF_COOKIE_NAME	'csrftoken'
STATICFILES_STORAGE	'django.contrib.staticfiles.storage.StaticFilesStorage'
EMAIL_SSL_CERTFILE	None
EXPLORER_CONNECTIONS	{'scenario_db': 'scenario_db'}
CSRF_FAILURE_VIEW	'django.views.csrf.csrf_failure'
MEDIA_ROOT	'C:\Users\MeSchoenbaum\Desktop\ADSM_9_16\media'
SESSION_COOKIE_PATH	'/'
CSRF_COOKIE_AGE	31449600
IGNORABLE_404_URLS	()
WEBPACK_LOADER	{'DEFAULT': {'BUNDLE_DIR_NAME': 'bundles/', 'STATS_FILE': 'C:\Users\MeSchoenbaum\Desktop\ADSM_9_16\webpack-stats.json'}}
SESSION_COOKIE_DOMAIN	None
LOGGING	{}
DEFAULT_CHARSET	'utf-8'
CRISPY_TEMPLATE_PACK	'bootstrap'
LOGOUT_URL	'/accounts/logout/'
OVERRIDE_DEBUG	False
USE_L10N	True
USE_TZ	True
MONTH_DAY_FORMAT	'F j'
SECURE_REDIRECT_EXEMPT	[]
FORMAT_MODULE_PATH	None
EMAIL_USE_SSL	False
X_FRAME_OPTIONS	'SAMEORIGIN'
USE_I18N	False
TIME_FORMAT	'P'
EXPLORER_CONNECTION_NAME	'scenario_db'
EMAIL_SUBJECT_PREFIX	'[Django] '
SESSION_FILE_PATH	None
FORCE_SCRIPT_NAME	None
EMAIL_PORT	25

[missyschoenbaum]

I should also note, some portion of this did work. Do I need to find the one that failed?

[ConradSelig]

Could you just drop my the entire file that failed? I have a sneaking suspicion about what happened but having the test case will still be useful

On Tue, Jun 25, 2019, 10:41 AM Missy Schoenbaum notifications@github.com wrote:

I should also note, some portion of this did work. Do I need to find the one that failed?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/NAVADMC/ADSM/issues/910?email_source=notifications&email_token=AHT6I27SL6UF2HV6UDHQH5LP4JDE3A5CNFSM4GFOQK32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYQ3MJI#issuecomment-505525797, or mute the thread https://github.com/notifications/unsubscribe-auth/AHT6I25BD77KONHCOIDAF6TP4JDE3ANCNFSM4GFOQK3Q .

[missyschoenbaum]

I'm going to email as csv's won't upload.

[missyschoenbaum]

Let me try again. Humm, the error we saw was exported that way. I tried a small sample and it worked with no problem. Let me try Texas and see what happens with it. I will also look at the data on the export error and see how we ended up with what we got in case I can give some extra details. Glad I tried it.

[missyschoenbaum]

After looking at the data, I have a clue to what may have happened. The Name field had commas in it. We don't restrict that. I think it was named like "This pdf applies to X,Y,Z" and then we grabbed X,Y,Z and split them apart. What do you think? Could we plan for that? What other special chars could kill us here?

[missyschoenbaum]

Another question - I'm working on documentation. When we get the new folder structure, are we going to have them put these into the Export folder to be Imported, or back at the scenario level.

[ConradSelig]

I like the commas theory, that makes a lot of sense. We can restrict comma usage, or try to convert the commas to something else when the user saves the function. We definitely need to do something about comma use however.

With the new folder structure files will import from the exports folder, I think some wording around the program will need to change so this makes sense. Do you want me to look for wording that might be confusing?

[BryanHurst]

Are the fields surrounded in quotes when exported?

[missyschoenbaum]

Do you want me to put this back in Current Sprint To do?

[ConradSelig]

Not yet surrounded in quotes... good idea however, I'l make the changes needed.

[ConradSelig]

Downside to this change, users will not be able to use single quotes in their function names (or any field for that matter) and the .csv files created are a little harder to hand-edit. The good news is that commas are good to go!

New formatting means function blocks will need to be re-exported before importing works. I put a check in place to prevent old-format function from being imported (as there was some funky action when I tried it). Unfortunately I cannot realistically support both formats.

@missyschoenbaum I'm emailing you the function block you sent me for testing of this bug that has been re-formatted to allow you to run it through the changes yourself.

[BryanHurst]

I'm going to take a look at this one as we should be able to handle quotes in the user content.

On Thu, Jun 27, 2019, 10:24 PM Conrad Selig notifications@github.com wrote:

Assigned #910 https://github.com/NAVADMC/ADSM/issues/910 to @BryanHurst https://github.com/BryanHurst.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/NAVADMC/ADSM/issues/910?email_source=notifications&email_token=AAJAWBL35HHGLH24VFJASLTP4WHAJA5CNFSM4GFOQK32YY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOSHG4QLI#event-2446182445, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJAWBJODEVOO6OWKAJ72P3P4WHAJANCNFSM4GFOQK3Q .

[BryanHurst]

@ConradSelig lets look into the csv writer's quoting capabilities.

csv.writer(output, quoting=csv.QUOTE_ALL)

I believe this will escape quotes in user strings automatically.

[ConradSelig]

We are also going to look at a selective importer. There is already a file selection browser in place for importing scenarios - we're going to see if we can piggyback off of that system for this purpose.

[ConradSelig]

Unfortunately even with the csv writer it is impossible to allow ALL characters to be used for naming functions. However the package did simplify my code so much that I was able to make the only restricted char "¿". Which you cannot even type on the keyboard!

I've provided two very complex test cases to try out, let me know if you have any problems with any of the functions you are importing!

CharTests.zip

I've still got more work to do on this ticket making the importer selective, so don't move to testing yet.

[ConradSelig]

With the decision at today's meeting to drop selective importing until a future update. This ticket is ready for testing. More changes to the functionality of the buttons will be made - but only in regard to where files are coming from and going to. Those changes will be in #953.

[missyschoenbaum]

Going to work in #953, closing this one.