Open waddella opened 3 years ago
duplicate of #3 ? closed #3
From the discussion today, it sounds like the goal is that code in the file is not automatically executed upon opening (specifically within the context of Windows as it's configured on the Reviewer's machine).
This gets a little tricky, since extensions are nothing more than cues to the operating system to deal with them in some specific way, and that behavior is often configured in user-space of the operating system (using the "Default Apps" menu). If the default app is a shell (windows command line, powershell, or even an R shell), then these would be automatically executed despite being ASCII text format files.
Based on our discussion today, it sounds like the HA reviewer machines have been configured to disallow software from automatically executing code based on an extension, but that .bat
files might be an exception, executing by default in the windows command line. This might be a notable exception to the ASCII text format rules.
It sounds like these are an exception to the more generalized ASCII guidance that we should include, and it would be great if any exceptions are narrowly defined. From the HA, it would be nice if we could get a list of file extensions which they would like to black-list (if there are any others aside from .bat
).
On linux systems I think it might be easier to draw this line based on the file's executable flag.
Would it be sufficient to say that this ASCII text recommendation should exclude files with extensions that are configured to launch and run in a shell in a fresh install of windows? I think this encompasses .bat
, and probably PowerShell (.ps1
, .ps2
+ a few others) as well, but I would need to get a hold of a Windows machine to be exhaustive.
ASCII should probably be supplanted with ASCII or UTF-8. Most modern languages, R and Python3 included, default to UTF8 when saving files.
As for .tar
as a collection of files, opens up a nice opportunity for containers (docker), as they can be saved and loaded as tarballs.
Note that they explicitly call out define.pdf
in which every example I've ever seen is UTF-8. Another place in the same document it notes:
3.3.5 Special Characters: Variables and Datasets
Variable names, as well as variable and dataset labels should include American Standard Code for Information Interchange (ASCII) text codes only. Variable values are the most broadly compatible with software and operating systems when they are restricted to ASCII text codes (printable values below 128). Use UTF-8 for extending character sets; however, the use of extended mappings is not recommended. Transcoding errors, variable length errors, and lack of software support for multi byte UTF-8 encodings can result in incorrect character display and variable value truncations. Ensure that LBSTRESC and controlled terminology extensions in LBTEST do not contain byte values 160-191 as some character mappings in that range may interfere with agency processes.
emphasis mine
Thus I think the idea is that avoid using UTF-8 and do NOT use in variable names/LBSTREC/LBTEST codelists.
A few other points about "binaries". The FDA uses binary format in many accepted files, the major one being PDFs (which often include zip type compression schemes internally). XPT files can contain floating points in IEEE (or IBM) format with big endian or little endian and that's definitely not ASCII text.
I suspect another reason for the deprecation of UTF-8 is fonts as the machines have limited installed fonts (times new roman, arial, courier new, symbol, and zapf dingbats; see the list in https://www.fda.gov/media/76797/download) so PDFs need to include fonts for anything but the approved ones.
In any event I think it's a conversation with a reviewer/CBER (or is it CDER?) IT.
Thanks @vmarenny & @kaz-gene-com
The .tar
option and the discussion about compressed pdf contents might open up a door for us to consider a .tar.gz
/.tgz
/.zip
, which are standard formats for distributing R packages (on unix
/mac
/windows
respectively).
Whether these are permitted as source code for a submission is one question (as they can be decompressed and extracted as plain-text files), and whether these files would be permissible in the submission portal is another open question. We discussed this option at today's meeting and plan to use the eCTD testing portal to get feedback on the viability of submitting a .tar.gz
/.tgz
/.zip
R package source code bundle.
This would make it much easier to install these packages on the receiving end of a submission, allowing a reviewer to simply use install.packages
instead of having to first build the packages themselves.
Thanks @vmarenny & @kaz-gene-com
The
.tar
option and the discussion about compressed pdf contents might open up a door for us to consider a.tar.gz
/.tgz
/.zip
, which are standard formats for distributing R packages (onunix
/mac
/windows
respectively).Whether these are permitted as source code for a submission is one question (as they can be decompressed and extracted as plain-text files), and whether these files would be permissible in the submission portal is another open question. We discussed this option at today's meeting and plan to use the eCTD testing portal to get feedback on the viability of submitting a
.tar.gz
/.tgz
/.zip
R package source code bundle.This would make it much easier to install these packages on the receiving end of a submission, allowing a reviewer to simply use
install.packages
instead of having to first build the packages themselves.
So is it acceptable to use R Source files like .tar.gz in submissions? Or do they have to be ASCII files?
I am copying over from:
https://github.com/RConsortium/submissions-wg/discussions/7
Motivation
Let's start the discussion about an alternative wording proposal for section
4.1.2.10
in the "FDA STUDY DATA TECHNICAL CONFORMANCE GUIDE" found here:https://www.fda.gov/media/143550/download
Currently (November 2020) the paragraph reads:
The goal is to be more specific about the file types that can be submitted as many have interpreted that paragraph that program files need to be submitted as
.txt
files.Am I correct that the actual constraint is that the files need to be:
iso-8859-1
encoding and notutf-8
.R
does not work but.r
would not workAlso would
.tar
be acceptable for a collection of ASCII files?Let's start the discussion here, I can update this first message with the agreed solution from the discussion thread.