Truncated resultset when no charset is specified

[palevi67]

I have a case where a "for rows.Next() {}" loop does not get all the rows. Curiously it only fails when no charset is specified, but with charset=None or charset=whatever it works correctly.

I include a test case:

func TestTruncatedSelect(t *testing.T) {
    conn, err := sql.Open("firebirdsql", "sysdba:masterkey@192.168.246.100/localsvn/hbde/hbde.fdb") // adding "?charset=None" makes the test pass ??
    require.NoError(t, err)

    // Update a table field with the Spanish word "sueño" (dream) codified as win1252
    dream, err := charmap.Windows1252.NewEncoder().String("sueño")
    require.NoError(t, err)
    stmt, err := conn.Prepare(`UPDATE T2 SET TEXTO_NN=? WHERE ENTERO_NN=2`)
    require.NoError(t, err)
    _, err = stmt.Exec(dream)
    require.NoError(t, err)

    // Expected number of rows
    var expected int
    err = conn.QueryRow(`SELECT count(*) FROM T2`).Scan(&expected)
    require.NoError(t, err)

    // Returned number of rows
    var actual int
    rows, err := conn.Query(`SELECT '<'||TEXTO_NN||'>' FROM T2 ORDER BY TEXTO_NN`)
    require.NoError(t, err)
    for rows.Next() {
        require.NoError(t, rows.Err())
        actual++
    }
    require.NoError(t, rows.Err())       // <-- opFetchResponse:Internal Error

    // Should match
    require.Equal(t, expected, actual)
}

--- FAIL: TestTruncatedSelect (0.32s)
    h:\hbde\firebird_test.go:235: 
            Error Trace:    h:/hbde/firebird_test.go:235
            Error:          Received unexpected error:
                            opFetchResponse:Internal Error
            Test:           TestTruncatedSelect

Tested in FB 2.5.9 + W7 and FB 5.0.0 + W11. Database charset is NONE and test table is:

-- DROP TABLE T2;

CREATE TABLE T2 (
    ENTERO_NN INTEGER NOT NULL,
    ENTERO INTEGER,
    TEXTO_NN VARCHAR(30) NOT NULL,
    TEXTO VARCHAR(3000),
    FECHA_NN DATE NOT NULL,
    FECHA DATE,
    HORA_NN TIME NOT NULL,
    HORA TIME,
    MOMENTO_NN TIMESTAMP NOT NULL,
    MOMENTO TIMESTAMP,
    MEMO BLOB SUB_TYPE TEXT,
    BINARIO BLOB SUB_TYPE BINARY,
    SIMPLE_NN FLOAT NOT NULL,
    SIMPLE FLOAT,
    DOBLE_NN DOUBLE PRECISION NOT NULL,
    DOBLE DOUBLE PRECISION,
    LETRAS_NN CHAR(30) NOT NULL,
    LETRAS CHAR(30),
    CONSTRAINT PK_T2 PRIMARY KEY (ENTERO_NN)
);

INSERT INTO T2
(ENTERO_NN, ENTERO, TEXTO_NN, TEXTO, FECHA_NN, FECHA, HORA_NN, HORA, MOMENTO_NN, MOMENTO, MEMO, BINARIO, SIMPLE_NN, SIMPLE, DOBLE_NN, DOBLE, LETRAS_NN, LETRAS)
VALUES(1, 1, 'uno', 'uno', '2024-06-04', '2024-06-04', '12:50:00', '12:50:00', '2024-06-04 12:50:00', '2024-06-04 12:50:00', 'memo', NULL, 1234.0, 1234.0, 12345678, 12345678, 'HOLA', 'ESCAROLA');

INSERT INTO T2
(ENTERO_NN, ENTERO, TEXTO_NN, TEXTO, FECHA_NN, FECHA, HORA_NN, HORA, MOMENTO_NN, MOMENTO, MEMO, BINARIO, SIMPLE_NN, SIMPLE, DOBLE_NN, DOBLE, LETRAS_NN, LETRAS)
VALUES(2, NULL, 'dos', NULL, '2024-06-04', NULL, '12:50:00', NULL, '2024-06-04 12:50:00', NULL, NULL, NULL, 1234.0, NULL, 12345678, NULL, 'HOLA', NULL);

INSERT INTO T2
(ENTERO_NN, ENTERO, TEXTO_NN, TEXTO, FECHA_NN, FECHA, HORA_NN, HORA, MOMENTO_NN, MOMENTO, MEMO, BINARIO, SIMPLE_NN, SIMPLE, DOBLE_NN, DOBLE, LETRAS_NN, LETRAS)
VALUES(4, 1, 'cuatro', '44444', '2024-06-04', '2024-06-04', '12:50:00', '12:50:00', '2024-06-04 12:50:00.000', '2024-06-04 12:50:00.000', 'memo', '69411602018583925187139511906871882297', 1234.0, 1234.0, 12345678, 12345678, 'ADIOS', NULL);

Is there any difference between charset=None and no charset?

[bat22]

@palevi67 you should check rows.Err() after loop.

https://pkg.go.dev/database/sql#Rows.Next

Next prepares the next result row for reading with the Rows.Scan method. It returns true on success, or false if there is no next result row or an error happened while preparing it. Rows.Err should be consulted to distinguish between the two cases.

[palevi67]

Yes @bat22, you're right! I have consequently updated the above test, and this time I get the following error:

opFetchResponse:Internal Error

[nakagami]

Do not encode to a specific character encoding.

In the above example

dream := "sueño"

instead of

dream, err := charmap.Windows1252.NewEncoder().String("sueño")

[palevi67]

Yes, this way is working, thought in this case "sueño" gets encoded as UTF8 ...

I guess that if you don't specify any encoding in the URL then UFT8 is assumed. And that is why you get an "Internal error" when data don't represent a valid UTF8 sequence.

I'm I right?

[nakagami]

I think the driver can handle utf-8 encoded strings, which is good enough.

when you want to pass Windows 1252 encoded strings?

[nakagami]

Not knowing which encoding the parameter string is, when converting it to a byte slice for query parameter

so

• Use the value of the charset of the connection parameter
• If charset is not specified, utf-8 is assumed

You may get the result you want if you pass a byte string as a parameter.

[palevi67]

Yes, this way is working, thought in this case "sueño" gets encoded as UTF8 ...

I seems to me that if you don't specify any encoding in the URL then UFT8 is assumed. And that is why you get an "Internal error" when data don't represent a valid UTF8 sequence.

I'm I right?

[palevi67]

when you want to pass Windows 1252 encoded strings?

I'm working in an windows environment with a WIN1552 which passes the data to other WIN1252 app, so no encoding seems the natural choice. Adding encoding=None to the solves my problem, I just didn't notice that the default encoding is UTF8 ...

Sorry for the trouble ...

[nakagami]

When retrieving a string from a database with charset=None the driver does not know if it is WIN1552 encoded or not.