ministero-salute / dcc-utils

Set of utilities to read EU Digital COVID Certificates, verify signatures and rules.
GNU Affero General Public License v3.0
65 stars 16 forks source link

zlib decompression fails with 'third dose' green passes #28

Closed onevodev closed 2 years ago

onevodev commented 2 years ago

DESCRIPTION: On decryption of the embedded zlib archive of green passes of people who has had their QR code re-emitted after their third dose of vaccine, the call on zlib fails to decompress the package, blocking the successive process of validation.

Steps to reproduce the behavior:

Expected behavior As per usual working behavior, i usually received the values i requested from my validator server: an object from where i extract name (obj.person), surname (same), DoB (obj.date_of_birth) and GP validity (obj.code). Here an example without personal info:

Is this DCC valid? true
Details:
VALID
true
string 

when decrypting a 'third dose' greenpass, i get this:

CertificateParsingError: certificate can't be parsed, incorrect data check
    at Object.fromRaw (/home/user/greenpass-sdk/src/certificate.js:121:11)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Server.<anonymous> (/home/user/greenpass-sdk/validatorServer.js:49:19) {
  data: {
    error: Error: incorrect data check
        at Zlib.zlibOnError [as onerror] (node:zlib:190:17)
        at processChunkSync (node:zlib:456:12)
        at zlibBufferSync (node:zlib:178:12)
        at Object.syncBufferWrapper [as inflateSync] (node:zlib:791:14)
        at Function.fromRaw (/home/user/greenpass-sdk/node_modules/dcc-utils/src/dcc.js:17:25)
        at Object.fromRaw (/home/user/greenpass-sdk/src/certificate.js:118:27)
        at Server.<anonymous> (/home/user/greenpass-sdk/validatorServer.js:49:37)
        at Server.emit (node:events:520:28)
        at parserOnIncoming (node:_http_server:951:12)
        at HTTPParser.parserOnHeadersComplete (node:_http_common:128:17) {
      errno: -3,
      code: 'Z_DATA_ERROR'
    }
  }
}

Screenshots error code

Food for Thoughts It looks like a zlib decompressing error, but as I'm not directly involved in the green pass creation I can't figure out what exactly changes in a newly-emitted GP. The error, i think, resides in the line 17 - dcc._coseRaw = zlib.inflateSync(base45Data); It looks like, by printing out the variable base45data in dcc.js the new GPs are longer byte strings, so it may be related to that, but can't tell for sure The process breaks right on that line and does not go on. Tried on amd64 and aarch64, same issue. I've seen zlib compilation issues on Apple M1 but experimenting on 64bit it seems like the issue remains the same. Perhaps passing an option to the method zlib.inflateSync as per https://nodejs.org/api/zlib.html#zlibinflatesyncbuffer-options might fix the problem, but i only managed to try unsuccessfully.

Please let me know if this is the wrong place to open the issue, but i was really unsure about where to post it, and this seemed the most reasonable place. Thanks for eventual aid fixing this. Also, a huge thanks to all developers i am using code from!

anversoft commented 2 years ago

Hi @onevoprojects, what a great explanation! We need some information to help you:

This is the code of my server (it is the old version, now I have also added cors and static files) and it works.

const app = require('./server.js');
const verifier = require('./verifier.js');
const schedule = require('node-schedule');
const port = process.env.SERVER_PORT || 3000;

function startCron () {
    console.log('CRON Started');
    schedule.scheduleJob('0 */15 * * * *', async function () {
        verifier.updateDefinitions();
    });
}

app.listen(port, async () => {
    console.log('ValidatorServer ready for requests, ');
    verifier.updateDefinitions(() => startCron(), () => startCron());
});
const express = require('express');
const helmet = require('helmet');
const indexRoutes = require('./routes/index.js');
const app = express();

// ============================================================================
// HELMET
// ============================================================================
app.use(helmet());
app.use(helmet.contentSecurityPolicy());
app.use(helmet.dnsPrefetchControl());
app.use(helmet.expectCt());
app.use(helmet.frameguard());
app.use(helmet.hidePoweredBy());
app.use(helmet.hsts());
app.use(helmet.ieNoOpen());
app.use(helmet.noSniff());
app.use(helmet.permittedCrossDomainPolicies());
app.use(helmet.referrerPolicy());
app.use(helmet.xssFilter());
// app.use(cors()); -- I haven't committed the latest version yet
// ============================================================================

// ============================================================================
// ROUTES
// ============================================================================
app.use('/', indexRoutes);
// ============================================================================

module.exports = app;

Routes

const express = require('express');
const verifier = require('../verifier.js');
const router = express.Router();
const root = require('../../rootdirectory.js');
const path = require('path');
const shell = require('shelljs');
const pck = require('../../package.json');

// Server info
router.get('/api/v1/check', function (req, res, next) {
    res.json({
        version: pck.version,
        description: pck.description,
        lastUpdate: verifier.data.lastDefinitionUpdate
    });
});

// Check the GP
router.get('/api/v1/checkdgc', async function (req, res, next) {
    await verifier.testDGC(req.query, res, verifier.data).catch(err => next(err));
});

// Update the definitions manually
router.post('/api/v1/updateDefinitions', async function (req, res, next) {
    verifier.updateDefinitions(val => res.send(val), err => next(err));
});

// Restart
router.post('/api/v1/serverRestart', function (req, res, next) {
    res.send('Server Restarting...');
    const rt = path.join(root, 'upgrade.sh');
    shell.exec(rt);
});

module.exports = router;

And then the validator

/**
 * Testa il green pass
 * @param {*} query Query della richiesta GET
 * @param {*} res Risposta del server
 * @param {*} data Data
 */
async function testDGC (query, res, data) {
    const ret = {
        message: undefined,
        valid: false,
        hash: undefined,
        surname: undefined,
        forename: undefined,
        error: undefined
    };

    // * Codice QR Green Pass
    const dgc = query.dgc;

    // ? Controlla che le definizioni siano aggiornate
    if (data.lastDefinitionUpdate === null || data.lastDefinitionUpdate === '') {
        // ! Le definizioni non sono aggiornate
        ret.error = 'DEFINIZIONI NON AGGIORNATE';
        res.status(400);
        res.json(ret);
    } else if (dgc === undefined) {
        // ! Il green pass non è valido
        ret.error = 'GP NON VALIDO';
        res.status(400);
        res.json(ret);
    } else if (query.validationMode === undefined || Number.isNaN(query.validationMode) || query.validationMode < 0) {
        // ! Il metodo di validazione non è valido
        ret.error = 'METODO DI VALIDAZIONE NON VALIDO';
        res.status(400);
        res.json(ret);
    } else {
        try {
            // * Tipo di validazione
            const validationMode = getValidationMode(query.validationMode);

            // * Crea il DCC
            const myDCC = await Certificate.fromRaw(dgc);

            // * Imposta il nome, cognome e Hash
            ret.forename = myDCC.person.givenName;
            ret.surname = myDCC.person.familyName;
            ret.hash = crypto.createHash('md5').update(myDCC.person.standardisedGivenName + myDCC.person.standardisedFamilyName + myDCC.dateOfBirth).digest('hex');

            // * Valida il Green Pass
            const validationResult = await Validator.validate(myDCC, validationMode);

            // * Prendi il risultato
            ret.valid = validationResult.result;

            // * Genera un messaggio di risposta
            switch (validationResult.code) {
            case Validator.codes.VALID:
                ret.message = 'VALIDO IN ITALIA ED EUROPA';
                break;
            case Validator.codes.TEST_NEEDED:
                ret.message = 'TEST RICHIESTO';
                break;
            case Validator.codes.NOT_VALID:
                ret.message = 'NON VALIDO';
                break;
            case Validator.codes.NOT_VALID_YET:
                ret.message = 'NON ANCORA VALIDO';
                break;
            case Validator.codes.REVOKED:
                ret.message = 'GREEN PASS REVOCATO';
                break;
            case Validator.codes.NOT_EU_DCC:
                ret.message = 'GREEN PASS NON EUROPEO';
                break;
            default:
                ret.error = validationResult.message;
                break;
            }
        } catch (error) {
            ret.valid = false;
            ret.error = error.data !== undefined ? error.data.error.message : error.message;
        }

        if (ret.valid) res.status(200);
        else res.status(400);

        res.json(ret);
    }
}
onevodev commented 2 years ago

Hello @anversoft , thanks for your answer and precious code. Sorry I forgot to explain the environment here. Running on RPi 3B+ w/ raspbian 64bit kernel, node version is currently

node -v && npm -v
v16.14.0
8.3.1

tried downgrading to

v14.19.0
6.14.16

but it didn't change a thing, maybe worsened it as it needed compiling for certain modules, and I'm on a RPi. As fro the SDK version, I am currently running on the latest git pulled from the repo, after having tried unsuccessfully the npm i installed version. I don't really think it's a versioning issue.

I have indeed verified that the string is passed correctly, both by comparing them out manually and because it still works with any other type of GP. Also tried comparing a serial-scanned GP string with one scanned with an Android qr reader for formatting issues but it is indeed the same. Would formatting other than utf-8 make a difference?

What made me think this matter is worth of an issue and not just my personal problem with hardware/string formatting is that i can't understand why this code would work with only certain kind of GPs, since the formatting happens in python (reading from serial USB scanner) and gets sent in the same way to the validator via http. My suspect is something may have changed in the 'third dose' edition, that breaks my code. I know they have been re-emitted by the government, but can't figure out any difference. That, or string formatting. I'm working on it right now, if you have any ideas..

Also digging into zlib decompression I have found that this elusive Z_DATA_ERROR comes from

inflateSync returns Z_OK if a possible full flush point has been found, Z_BUF_ERROR if no more input was provided, Z_DATA_ERROR if no flush point has been found, or Z_STREAM_ERROR if the stream structure was inconsistent. In the success case, the application may save the current current value of total_in which indicates where valid compressed data was found. In the error case, the application may repeatedly call inflateSync, providing more input each time, until success or end of the input data.

But as said previously, changing the behavior of flushing does not fix the error

astagi commented 2 years ago

Hi @onevoprojects I think you have some parsing errors when you get your parameter from url, read this https://github.com/italia/verificac19-sdk/issues/4#issuecomment-968832996

Also I suggest to pass your raw DCC using a POST request, if possible.

onevodev commented 2 years ago

Yes. Thank you - After hours of trial and error I am finding that the issue mainly resides in the encoding of the gp string between python and nodejs. I am still struggling to find a proper way to univocally format a string to move between programs, but at least I know the root of the problem. Thank you again for your support. I still think there is something deeply wrong with either GP encoding or zlib, but I think the case is solved regarding dcc-utils itself.

astagi commented 2 years ago

Sounds great @onevoprojects ! If you want to keep us updated with your progress on this thread feel free to update it, for now I'll close this issue.

onevodev commented 2 years ago

Well, it's been a hard day's night but here we are: The issue originated (mainly) from python's trailing characted in serial-read strings, which added a funny &0D at the end of (some?) GP strings. For anybody else who might encounter the almighty zlib's Z_DATA_ERROR, here's the code that fixed it for me:

    gp_raw = None; gp = None # initialize
    # start reading from serial...
    sp.flushInput() # clear previous buffer
    gp_raw = sp.readline().decode('utf-8')
    # wait for proper message to come in1
    if gp_raw != '':

        # magic formatting
        gp_raw = gp_raw.strip()
        gp = {'dgc': gp_raw}
        gp = urllib.parse.urlencode(gp)

        # reset raw variable
        gp_raw = None

where gp_raw is the raw qr code and gp becomes the GET request (not POST) main parameter. Again, my deepest thanks for being so helpful, @anversoft @astagi

astagi commented 2 years ago

Glad to be helpful @onevoprojects 🙌