Merging #250 and deploying to staging uncovered certain issues.
Unpacking attachments fails on the agent used for testing because unpacking uses the filter argument of tarfile.extractall which is not supported in the Python version installed on the agent, i.e. 3.8.10.
This issue would normally have surfaced in the agent testing workflow, prior to deployment. It did not because that workflow does not specify a Python patch version (only 3.8), resulting in the latest patch version being installed, i.e. 3.8.18). The filter argument was added in version 3.8.17, so it is supported in the environment used for testing in the workflow but not in the actual agent.
Testing on staging revealed that the provisioning and firmware update stages were triggered when attachments were provided under the provision_data and firmware_update_data fields. The code assumes that any data under these fields suggests that the corresponding stages should be triggered.
Changes
A fully qualified Python version (3.8.12) is specified in the agent testing workflow. It does not match the Python version installed on the agent because that version is not available on our self-hosted runners, but it is sufficient for testing because the filtering feature was introduced in 3.8.17.
The filter functionality provides an important safety feature. Instead of re-implementing what would necessarily be a subset of that functionality, I decided to use a minimal subset of the implementation provided in the main branch of the Python standard library. A check in the imports section decides whether or not to use the tarfile that comes with the current Python installation (if it supports filter) or the alternative one otherwise. The major advantages to this solution is that (a) the actual unpacking code remains unchanged and uses the filter functionality and (b) this is not a version-sensitive solution, i.e. whenever it can be guaranteed that the agents will run a Python version supporting filter, the unpacking code will remain the same and only the conditional import will be removed.
After attachments are unpacked, attachment data is stripped from the job data dict, restoring it to a state that is identical to how it would be if attachments had not been included (i.e. like the good old days). This respects any assumptions that Testflinger makes about the contents of the job data dict and removes any possible interference with other stages. Note that attachment data doesn't even need to be included in the job data dict in the first place (it's all in the gzipped-tar archive) and it may be removed anyway in a later version. The reasons it has been kept are that (a) it is useful to display the attachments under the Provision Data and Firmware update data sections of the Job page on Testflinger and (b) it is yet unclear how that attachment data might be used by a device connector later down the line.
Description and resolved issues
Merging #250 and deploying to staging uncovered certain issues.
filterargument oftarfile.extractallwhich is not supported in the Python version installed on the agent, i.e.3.8.10.3.8), resulting in the latest patch version being installed, i.e.3.8.18). Thefilterargument was added in version3.8.17, so it is supported in the environment used for testing in the workflow but not in the actual agent.provision_dataandfirmware_update_datafields. The code assumes that any data under these fields suggests that the corresponding stages should be triggered.Changes
3.8.12) is specified in the agent testing workflow. It does not match the Python version installed on the agent because that version is not available on our self-hosted runners, but it is sufficient for testing because the filtering feature was introduced in3.8.17.filterfunctionality provides an important safety feature. Instead of re-implementing what would necessarily be a subset of that functionality, I decided to use a minimal subset of the implementation provided in themainbranch of the Python standard library. A check in the imports section decides whether or not to use thetarfilethat comes with the current Python installation (if it supportsfilter) or the alternative one otherwise. The major advantages to this solution is that (a) the actual unpacking code remains unchanged and uses thefilterfunctionality and (b) this is not a version-sensitive solution, i.e. whenever it can be guaranteed that the agents will run a Python version supportingfilter, the unpacking code will remain the same and only the conditional import will be removed.Provision DataandFirmware update datasections of the Job page on Testflinger and (b) it is yet unclear how that attachment data might be used by a device connector later down the line.