Propose & implement approach for secrets

[joewiz]

where needed:

• s3: used on http://localhost:8080/exist/apps/hsg-shell/historicaldocuments/ebooks and FRUS volume landing pages (the latter not yet implemented in new site)
• twitter: used currently on homepage
• tumblr: used currently on homepage
• eXist user account info: used in our ant scripts for populating an empty database and uploading/deleting content via oXygen

[joewiz]

we cannot launch until twitter and tumblr self-update, and until users can trigger s3 polling on select resources.

[JoernT]

secrets are hardcoded within queries - this should be changed. A simple solution would be an Ant property file to keep the secrets and use token filtering to substitute them while building.

[JoernT]

could you point out where in the code the relevant secrets are stored? A quick search didn't reveal this for me.

[joewiz]

e.g., in paho/trunk:

/db/cms/apps/social-media/modules/tumblr-client.xq

[joewiz]

Having just re-deployed hsg-shell to the beta site (it's looking great! https://history.state.gov/beta/), I noticed another step that is currently manual:

• When we need to "git pull" changes to the build.properties (in /data/history.state.gov/hsg-project), local changes to this file have to be discarded to receive the update, but then the local changes have to be reapplied. specifically, the admin password has to be entered here so that the ant deploy can successfully put the xars into the db. and while the site remains in the /beta URL space, the xmlrpc address has to be modified.
• I have to open eXide's configuration.xml and disable guest access by changing <restrictions guest="yes"/> to no. It would be nice if we could apply this setting prior to uploading the .xar, and avoid a period when eXide is exposed to guests.

Also, when we wipe the db, we have to re-add the admin user password via client.sh. This is probably unavoidable unless we institute a "backup/restore" system, but it is an aspect of the "secrets" issue.

[joewiz]

Also, when we rebuild eXist, we need to remember to delete the contents of the autodeploy directory; otherwise we won't get the latest version of the dashboard app. Or, since hsg-project already builds and deploys the latest versions of eXide and shared-resources, should we add dashboard to this list?

[joewiz]

Also, starting from a "vanilla" build of eXist that is periodically updated from source control and rebuilt, we need a way of activating certain settings - like memory settings and which extension modules need to be built or disabled. (Note for future: memory settings are stored in bin/functions.d/eXist-settings.sh. Until just now we've always been running this beta site with -Xmx1024m; now, it's 6144m to match the main hsg production site.)

[joewiz]

Also, in posting new or changed files to the production server, we rely on an ant script (called from oXygen's External Tools menu) to perform the XML-RPC upload. This script pulls credentials for a write-access user from a properties file, which we currently store in subversion. Until we move to the hypothetical continuous-integration based deployment method (in which our version control repository triggers web hooks to push newly committed files to the production server), our ant script will need to be updated for use with hsg-shell's new architecture - and it will need a way to make use of credentials for the user for pushing the file.

[joewiz]

Here are the current ant targets:

    <!-- ================================================= -->
    <!-- Targets for uploading files from oXygen to the db -->
    <!-- ================================================= -->

    <target name="upload-current-file-to-localhost" description="Uploads the working copy file currently open in oXygen XML Editor to eXist-db localhost">
        <property name="parent-directory" value="${arg1}"/>
        <property name="file-name" value="${arg2}"/>
        <property name="file-path" value="${parent-directory}/${file-name}"/>
        <echo message="Uploading ${file-name} from ${parent-directory} to eXist-db localhost"/>
        <path id="parent-dir">
            <pathelement path="${parent-directory}"/>
        </path>
        <fileset file="${file-path}" id="file"/>
        <pathconvert targetos="unix" property="file-path.unix" refid="parent-dir">
            <map from="c:" to=""/>
        </pathconvert>
        <if>
            <contains string="${file-path.unix}" substring="/db/" />
            <then>
                <propertyregex property="target-collection" input="${file-path.unix}"
                    regexp=".*?/db/(.*)"
                    select="\1"
                    defaultvalue="Error"
                    override="true"/>
                <property name="target-xmldb-collection-uri" value="${db.xmldb.uri}${target-collection}"/>
                <xdb:store user="${db.xmldb.username}" password="${db.xmldb.password}"
                    uri="${target-xmldb-collection-uri}" createcollection="true" createsubcollections="true">
                    <fileset refid="file"/>
                </xdb:store>
            </then>
            <else>
                <echo>ERROR: The directory ${parent-directory} is not from the working copy directory of files for the eXist-db database</echo>
            </else>
        </if>
    </target>

    <target name="upload-current-file-to-hsg" description="Uploads the working copy file currently open in oXygen XML Editor to eXist-db on history.state.gov">
        <property name="parent-directory" value="${arg1}"/>
        <property name="file-name" value="${arg2}"/>
        <property name="file-path" value="${parent-directory}/${file-name}"/>
        <echo message="Uploading ${file-name} from ${parent-directory} to eXist-db on history.state.gov"/>
        <path id="parent-dir">
            <pathelement path="${parent-directory}"/>
        </path>
        <fileset file="${file-path}" id="file"/>
        <pathconvert targetos="unix" property="file-path.unix" refid="parent-dir">
            <map from="c:" to=""/>
        </pathconvert>
        <if>
            <contains string="${file-path.unix}" substring="/db/" />
            <then>
                <propertyregex property="target-collection" input="${file-path.unix}"
                    regexp=".*?/db/(.*)"
                    select="\1"
                    defaultvalue="Error"
                    override="true"/>
                <property name="target-xmldb-collection-uri" value="${db.production.xmldb.uri}${target-collection}"/>
                <xdb:store user="${db.production.xmldb.username}" password="${db.production.xmldb.password}"
                    uri="${target-xmldb-collection-uri}" ssl="true" createcollection="true" createsubcollections="true">
                    <fileset refid="file"/>
                </xdb:store>
            </then>
            <else>
                <echo>ERROR: The directory ${parent-directory} is not from the working copy directory of files for the eXist-db database</echo>
            </else>
        </if>
    </target>

    <target name="delete-current-file-from-localhost" description="Deletes the working copy file currently open in oXygen XML Editor from eXist-db localhost">
        <property name="parent-directory" value="${arg1}"/>
        <property name="file-name" value="${arg2}"/>
        <property name="file-path" value="${parent-directory}/${file-name}"/>
        <echo message="Deleting ${file-name} from eXist-db localhost"/>
        <path id="parent-dir">
            <pathelement path="${parent-directory}"/>
        </path>
        <fileset file="${file-path}" id="file"/>
        <pathconvert targetos="unix" property="file-path.unix" refid="parent-dir">
            <map from="c:" to=""/>
        </pathconvert>
        <if>
            <contains string="${file-path.unix}" substring="/db/" />
            <then>
                <propertyregex property="target-collection" input="${file-path.unix}"
                    regexp=".*?/db/(.*)"
                    select="\1"
                    defaultvalue="Error"
                    override="true"/>
                <property name="target-xmldb-collection-uri" value="${db.xmldb.uri}${target-collection}"/>
                <echo>xdb:remove user="${db.xmldb.username}" password="${db.xmldb.password}"
                    uri="${target-xmldb-collection-uri}" resource="${file-name}"</echo>
                <xdb:remove user="${db.xmldb.username}" password="${db.xmldb.password}"
                    uri="${target-xmldb-collection-uri}" resource="${file-name}">
                </xdb:remove>
            </then>
            <else>
                <echo>ERROR: The directory ${parent-directory} is not from the working copy directory of files for the eXist-db database</echo>
            </else>
        </if>
    </target>

    <target name="delete-current-file-from-hsg" description="Deletes the working copy file currently open in oXygen XML Editor from eXist-db on history.state.gov">
        <property name="parent-directory" value="${arg1}"/>
        <property name="file-name" value="${arg2}"/>
        <property name="file-path" value="${parent-directory}/${file-name}"/>
        <echo message="Deleting ${file-name} from eXist-db on history.state.gov"/>
        <path id="parent-dir">
            <pathelement path="${parent-directory}"/>
        </path>
        <fileset file="${file-path}" id="file"/>
        <pathconvert targetos="unix" property="file-path.unix" refid="parent-dir">
            <map from="c:" to=""/>
        </pathconvert>
        <if>
            <contains string="${file-path.unix}" substring="/db/" />
            <then>
                <propertyregex property="target-collection" input="${file-path.unix}"
                    regexp=".*?/db/(.*)"
                    select="\1"
                    defaultvalue="Error"
                    override="true"/>
                <property name="target-xmldb-collection-uri" value="${db.production.xmldb.uri}${target-collection}"/>
                <xdb:remove user="${db.production.xmldb.username}" password="${db.production.xmldb.password}"
                    uri="${target-xmldb-collection-uri}" resource="${file-name}" ssl="true">
                </xdb:remove>
            </then>
            <else>
                <echo>ERROR: The directory ${parent-directory} is not from the working copy directory of files for the eXist-db database</echo>
            </else>
        </if>
    </target>

Our oXygen external tools menu calls these via the following settings in our oxygen .xpr file:

<xsltCustomEngines>
    <field name="outEncoding">
        <null/>
    </field>
    <field name="outContentType">
        <null/>
    </field>
    <field name="errEncoding">
        <null/>
    </field>
    <field name="cmd">
        <String>sh "${oxygenInstallDir}/tools/ant/bin/ant" upload-current-file-to-localhost -Darg1="${cfd}" -Darg2="${cfne}" -Dlog4j.properties="${pd}/build/log4j.properties"</String>
    </field>
    <field name="description">
        <String>Upload the currently open working copy file to eXist-db on localhost 8080</String>
    </field>
    <field name="name">
        <String>upload-current-file-to-localhost</String>
    </field>
    <field name="workingDirectory">
        <String>${pd}/build</String>
    </field>
    <field name="keyStroke">
        <null/>
    </field>
    <field name="isBuiltin">
        <Boolean>false</Boolean>
    </field>
    <field name="showConsole">
        <Boolean>true</Boolean>
    </field>
</xsltCustomEngines>
<xsltCustomEngines>
    <field name="outEncoding">
        <null/>
    </field>
    <field name="outContentType">
        <null/>
    </field>
    <field name="errEncoding">
        <null/>
    </field>
    <field name="cmd">
        <String>sh "${oxygenInstallDir}/tools/ant/bin/ant" upload-current-file-to-hsg -Darg1="${cfd}" -Darg2="${cfne}" -Dlog4j.properties="./log4j.properties"</String>
    </field>
    <field name="description">
        <String>Upload the currently open working copy file to eXist-db on history.state.gov</String>
    </field>
    <field name="name">
        <String>upload-current-file-to-hsg</String>
    </field>
    <field name="workingDirectory">
        <String>${pd}/build</String>
    </field>
    <field name="keyStroke">
        <null/>
    </field>
    <field name="isBuiltin">
        <Boolean>false</Boolean>
    </field>
    <field name="showConsole">
        <Boolean>true</Boolean>
    </field>
</xsltCustomEngines>
<xsltCustomEngines>
    <field name="outEncoding">
        <null/>
    </field>
    <field name="outContentType">
        <String>text/plain</String>
    </field>
    <field name="errEncoding">
        <null/>
    </field>
    <field name="cmd">
        <String>sh "${oxygenInstallDir}/tools/ant/bin/ant" delete-current-file-from-localhost -Darg1="${cfd}" -Darg2="${cfne}" -Dlog4j.properties="${pd}/build/log4j.properties"</String>
    </field>
    <field name="description">
        <String>Delete the currently open working copy file from eXist-db on localhost 8080</String>
    </field>
    <field name="name">
        <String>delete-current-file-from-localhost</String>
    </field>
    <field name="workingDirectory">
        <String>${pd}/build</String>
    </field>
    <field name="keyStroke">
        <null/>
    </field>
    <field name="isBuiltin">
        <Boolean>false</Boolean>
    </field>
    <field name="showConsole">
        <Boolean>true</Boolean>
    </field>
</xsltCustomEngines>
<xsltCustomEngines>
    <field name="outEncoding">
        <null/>
    </field>
    <field name="outContentType">
        <String>text/plain</String>
    </field>
    <field name="errEncoding">
        <null/>
    </field>
    <field name="cmd">
        <String>sh "${oxygenInstallDir}/tools/ant/bin/ant" delete-current-file-from-hsg -Darg1="${cfd}" -Darg2="${cfne}" -Dlog4j.properties="${pd}/build/log4j.properties"</String>
    </field>
    <field name="description">
        <String>Delete the currently open working copy file from eXist-db on history.state.gov</String>
    </field>
    <field name="name">
        <String>delete-current-file-from-hsg</String>
    </field>
    <field name="workingDirectory">
        <String>${pd}/build</String>
    </field>
    <field name="keyStroke">
        <null/>
    </field>
    <field name="isBuiltin">
        <Boolean>false</Boolean>
    </field>
    <field name="showConsole">
        <Boolean>true</Boolean>
    </field>
</xsltCustomEngines>

[windauer]

eXist Solutions will setup a private Git repository and use Pass (https://www.passwordstore.org/) to organize the passwords (Pass keeps each password in an extra text file, which is PGP encrypted). For a better user experience qtpass (https://qtpass.org/) can be utilized as graphical user interface. eXist Solutions will further provide a short documentation how to store and read passwords with qtpass / pass.

[joewiz]

From chat:

Me:

Hi, Lars, my question is generally how you suggest we enter credentials into the production system? Do we decrypt on the remote instance, or decrypt locally and paste in over SSH shells & file transfer clients?

@windauer:

As you prefer. But I would think, the easiest is to have Pass and the Gitlab repo on any machine where we need often passwords

Me:

The "paste in" method occurred to me this morning. Before that, I couldn't figure out how we would decrypt on the remote system - whose PGP key to use, for example? whose master passphrase? So the "paste in" method seems most straightforward. And you're right, we'll all have Pass and access to the Gitlab repo on the machines we're using to access the remote servers. That makes sense to me now. So I'll close the ticket, and will open some related tickets for items mentioned in that ticket that were not directly related to passwords.

@windauer:

yes, I got your point as well. Let me think about it please and have a chat with @tobi and @tasmo tomorrow morning. We could as well have shared private keys. Like "hsg-master" to decrypt anything and "hsg-social" only for facebook and twitter.. or any other categories, if we need these at all. A "group" would then share the same private key over all the machines, where the need it. But as I said, I would like to think about it for a moment longer ;)

So I'll keep this open pending @windauer's final ideas on our approach to storing & retrieving secrets.

[tasmo]

Solution: Using pass syncing passwords to a local machine.