Use in a GDPR-friendly way

[PierreMesure]

Hi,

Amazing work on this tool and kudos on making it as easy to use as possible!

As you maybe know, Firebase and Google Sheets are not usable for any research gathering personal data from EU citizens. Since Schrems II, it's been in practice illegal to use any service owned by a US company as the US legislation gives surveillance agencies a right to access the data.

I was then wondering if the possibility of saving the data locally as CSV or sending every answer to an e-mail address exists in this tool (I know it's still a part of htmlq but the project is abandoned. I couldn't really find it by testing the tool or looking in the code.

[shawnbanasick]

Thanks for the information. I'll have to look for a more GDPR-friendly database setup.

If you want to use email to have the participants report the Q sort results you can use these files until I can get enough time to update the Configurator: EQ web sort - email.zip

You can edit them with the current Configurator to set up your project, but then you'll have to manually edit a couple of entries in the config.xml file.

Guide: EQ web sort with Email Responses.pdf

Here is a working demo: https://eqwebsort-email.netlify.app

[shawnbanasick]

Here is a better version with a copy to clipboard button for backup eq web sort - email.zip

updated guide: EQ web sort with Email Responses (1).pdf

Here is a working demo: https://eqwebsort-email.netlify.app/

[PierreMesure]

Thank you Shawn, I haven't had time to look at it, will try this weekend!

[PierreMesure]

Hi Shawn,

I've now looked closer at the code and I think I understand what various settings do. I must say it could help with some documentation.

• email: The Submit button opens the device's e-mail client from which the data can then be sent
• local: The Submit button saves the data in the device's browser local storage

I know a static website is always going to be limited without using an external API like Firebase's or SteinHQ's but would there be a way to run this code on a little (Node, PHP?) server and save the data there as CSV or in a database?

That would be a solution as elegant as using Firebase but more privacy-friendly.

[PierreMesure]

Maybe a more concrete question would be: Is your project still compatible with the PHP script developed in the FlashQ project (last updated in 2007 it seems)? Or would you know about any newer alternative?

EDIT: The more I read, the more I feel like you removed that compatibility with this project and what I need to use is the last version of easy-htmlq.

[natalie427]

Hi

Maybe a more concrete question would be: Is your project still compatible with the PHP script developed in the FlashQ project (last updated in 2007 it seems)? Or would you know about any newer alternative?

EDIT: The more I read, the more I feel like you removed that compatibility with this project and what I need to use is the last version of easy-htmlq.

Hi Pierre,

Did you find an alternative either with using this software or the use of other software? I would like to use this software for my thesis but I suspect it will not meet the information governance expectations.

Thank you Natalie

[PierreMesure]

Hi Natalie, I ended up using htmlq. It's running with PHP on a Ubuntu server. We're running two versions in two different languages, see the result here.

It would be great to add the CSV function to the latest code as it looks better in terms of user-friendliness and probably accessibility and security as well (the code I used is from 2015).

[natalie427]

Hi Pierre,

Thank you!

---

From: Pierre @.> Sent: 11 April 2023 03:58 To: shawnbanasick/eq-web-sort @.> Cc: Natalie Softley @.>; Comment @.> Subject: Re: [shawnbanasick/eq-web-sort] Use in a GDPR-friendly way (Issue #4)

Hi Natalie, I ended up using htmlqhttps://github.com/aproxima/htmlq. It's running with PHP on a Ubuntu server. We're running two versions in two different languages, see the result herehttps://seafood.blandon.se/jp/.

It would be great to add the CSV function to the latest code as it looks better in terms of user-friendliness and probably accessibility and security as well (the code I used is from 2015).

— Reply to this email directly, view it on GitHubhttps://github.com/shawnbanasick/eq-web-sort/issues/4#issuecomment-1502941942, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A7DHJVM7S7BD55YDWPLNE3LXAUMKDANCNFSM6AAAAAAVOOLJW4. You are receiving this because you commented.Message ID: @.***>

[PierreMesure]

No worries, don't hesitate to reach out if you need help. I don't have time to open source the exact code we're using but I'll try to do it in the coming months.

[PierreMesure]

I found a few hours to make this project work with local file save as CSV and JSON.

I used ExpressJS to be consistent with your current stack @shawnbanasick. I didn't modify any of your code so right now, it uses the sheets setting with a custom URL ({domain}/save)

I also added a path to download the files with the right password in the URL ({domain}/download/csv?password=xxxxxxxxx).

@shawnbanasick, would you be open to accepting any of this as a PR? I could also provide one to add a setting that prevents Firebase scripts from loading altogether so that EU researchers can use your work in peace. 😊

app.js

const express = require('express');
const path = require('path');
const fs = require('fs');
const papa = require("papaparse");
const app = express();
const port = 9000;
const password = 'xxxxxxxxx'

function saveToFile(filename, newData) {
  fs.readFile(`${filename}.json`, 'utf8', (err, dataString) => {
    if (err) {
      console.log(`Could not find ${filename}. Creating an empty file.`);
      var data = [];
    } else {
      console.log("Opened the file")
      var data = JSON.parse(dataString);
    }

    data.push(newData);

    write(`${filename}.json`, JSON.stringify(data));
    write(`${filename}.csv`, papa.unparse(data));
  });
}

function write(filename, data) {
  fs.writeFile(filename, data, function(err) {
      if(err) {
          console.log(err);
      }
      console.log(`Writing ${filename}`);
  });
}

app.use(express.static(path.join(__dirname, '/')));
app.use(express.json());

app.get('/', (req, res) => {
  res.sendFile(path.join(__dirname, '/index.html'));
})

app.post('/save/Sheet1', (req, res) => {
    let data = req.body[0];
    let fixed_data = {};

    Object.keys(data).forEach(element => {
      let [key, value] = data[element].split(': ');
      fixed_data[key] = value;
    });

    if (fixed_data.sort != 'no_data') {
      let statementRankings = fixed_data.sort.split('|');

      statementRankings.forEach((element, i) => {
        fixed_data[`s${i}`] = element;
      });
    }

    saveToFile('data', fixed_data);

    res.send({});
})

app.get('/download/:format', (req, res) => {

  if (req.query.password && req.query.password == password) {
    res.download(path.join(__dirname, `/data.${req.params.format}`));
  }
  else {
    res.send('Password missing or incorrect.')
  }
})

app.listen(port, () => {
  console.log(`Starting the server on port ${port}`);
})

[shawnbanasick]

Thanks for the code. I think the best way to move forward is for me to fork the project and have a completely Google-free version that pulls in your code and keep this version for those who are OK to use Firebase. That's going to be much easier than trying to conditionally pull out the Google code with the Configurator setup process.

[PierreMesure]

I guess you mean a branch as you can't fork your own project. That's an option but I think it would be good to be able to choose CSV + server as an alternative to Firebase/Sheets through the configurator. So Ideally, everything could be configured in a unique release. Happy to help in any case! 😊

And tell me if you want me to start a PR that we can iterate on.

PS: The code I published above has two dependencies, ExpressJS and PapaParse. The latter can be removed in my opinion and replaced by a small function writing the CSV. Here's the package.json:

{
  "name": "eq-web-sort-v200",
  "version": "1.0.0",
  "description": "Server-side requirements for eq-web-sort",
  "main": "app.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "express": "^4.18.2",
    "papaparse": "^5.4.1"
  }
}

[joshmitcho-McMaster]

@PierreMesure Shawn has made a new repo for this. So far just stripped out the Google code, ready for changes https://github.com/shawnbanasick/eq-web-sort-dev

[PierreMesure]

Hi @shawnbanasick, I had a look at your latest release and the tutorial you wrote to deploy the site on Netlify. I think it's a great addition!

I want to reiterate that the original problem was until last Summer, the USA was considered an unsafe 3rd country in regards to GDPR and because of the US surveillance agencies prerogatives. That meant that any US-owned company was de facto not an option to store personal data as the US law give the NSA and CIA a right to access it without informing the data controllers. Not Google, neither Netlify, even if they claim that they are compliant in their terms and conditions. That situation changed a little when the US and the EU Commission entered an agreement called the Transatlantic Data Privacy Framework. But since the surveillance legislation of the US hasn't changed, it is likely that this agreement will be challenged and overturned in court like the two previous ones.

In this situation, this issue and the code I pasted above was meant to make the app easy to deploy on a standard server which could be located anywhere. And it is still relevant for someone who wants to avoid relying on one of the providers you support configuration for. 🙂

Thanks again for your work!

[joshmitcho]

@PierreMesure Did you end up forking the repo to add your changes? I'd really appreciate access to a version of this tool like you describe

[PierreMesure]

Hi @joshmitcho, I didn't but you can just reuse the two files I put above in this issue. Just save the first as app.js and the second as package.json. After that, install Node and run:

npm i
node app.js

[paulterinho]

@PierreMesure Could I ask what you have in your index.html file? The code bonks on this line...

app.get('/', (req, res) => {
  res.sendFile(path.join(__dirname, '/index.html'));
})

[PierreMesure]

Hi @paulterinho, I haven't tested this code in years so there might have been some changes.

But make sure you put it at the base of the generated folder which should have a file called index.html.

[paulterinho]

Hi @PierreMesure ,

Thanks for the update! I was thinking that this would have been too long ago. Our research team had to adopt another project because of the time-risk adapting (and GDPR privacy concerns bc of firebase) this project for a use-case it wasn't designed for.

That being said, I Just wanted to say, @shawnbanasick , 100% thank you for this project. I have other colleagues that use it the way it is supposed to be used (via binary installation), and they really appreciate the ENORMOUS amount of time you have taken out of your busy schedule to have created and maintain this project.

Thanks so much for both of your works!