leomarquine / php-etl

Extract, Transform and Load data using PHP.
MIT License
178 stars 81 forks source link

Boolean values appear to extract as strings #15

Closed dcramble closed 5 years ago

dcramble commented 5 years ago

I'm using the table extractor on a PostgreSQL 7.8 database and it appears Boolean values are being extracted as strings. I'm trying to load the data back into a PostgreSQL 9.3 database and PostgreSQL is complaining:

Invalid text representation: 7 ERROR: invalid input syntax for type boolean: ""

Do I have to create a transformer to convert these strings to bool, or is there some additional configuration options I missed?

dcramble commented 5 years ago

Looks like I may have been wrong about the Extractor. With further debugging, it appears the boolean values are extracted as bool, but they are trying to load as strings. Since this happens at the load phase, a transformer can't fix this issue. Is there another way to resolve this?

dcramble commented 5 years ago

I was able to work around this by creating a transformer to convert the extracted boolean values to strings of 'true' and 'false'. PostgreSQL 9 (at the least) knew how to process these to bool types when loading the data.

leomarquine commented 5 years ago

I just tested the table extractor on a PostgreSQL database (version 10.5) and the value is extracted as a boolean and it still is a boolean in the load step.

As it might be useful in many more situations, I'll add a type cast transformer in a future release.

hammerffallbk commented 5 years ago

Extracting data from postgres 9.5 and load to 10.6 postgres, the error is still there. Version of library 2.2.2

SQLSTATE[22P02]: Invalid text representation: 7 ERROR: invalid input syntax for type boolean: ""

Apparently it's cause PDOStatement::execute($params), every single param is treated as PDO::PARAM_STR

dcramble commented 5 years ago

If you need a work-around, I created a transformer to handle this:

<?php

namespace Marquine\Etl\Transformers;

use Marquine\Etl\Row;
use InvalidArgumentException;

class ConvertBoolean extends Transformer
{
    /**
     * Transformer columns.
     *
     * @var array
     */
    protected $columns = [];

    /**
     * Properties that can be set via the options method.
     *
     * @var array
     */
    protected $availableOptions = [
        'columns'
    ];

    /**
     * Transform the given row.
     *
     * @param  \Marquine\Etl\Row  $row
     * @return void
     */
    public function transform(Row $row)
    {
        $row->transform($this->columns, function ($column) {
            if (true == filter_var($column, FILTER_VALIDATE_BOOLEAN)) {
              return 'true';
            }
            else {
              return 'false';
            }
        });
    }

}
hammerffallbk commented 5 years ago

Thanks, I added the code to it, by hand, but it's not actually on the project.