search

filamentphp / filament

A collection of beautiful full-stack components for Laravel. The perfect starting point for your next app. Using Livewire, Alpine.js and Tailwind CSS.
https://filamentphp.com
MIT License
19.3k stars 2.96k forks source link

Importing a CSV file encoded with ISO-8859-1 #12063

Closed Rahmon closed 7 months ago

Rahmon commented 7 months ago

Package

filament/actions

Package Version

v3.2.60

Laravel Version

v10.48.4

Livewire Version

v3.4.9

PHP Version

PHP 8.3.3

Problem description

When I try to import a CSV file encoded with ISO-8859-1, the columns are not mapped although I set up it in the getColumns method.

public static function getColumns(): array
    {
        return [
            ImportColumn::make('name')
                ->guess(['Açaí'])
                ->requiredMapping()
                ->rules(['required', 'max:255']),
            ...
      ]
}

image

Even selecting the correct columns in the interface, I get the exception Unable to encode attribute [data] for model [Filament\Actions\Imports\Models\FailedImportRow] to JSON: Malformed UTF-8 characters, possibly incorrectly encoded.

Also, I noticed that the row data has the same issue. The expected output should be Conceição instead of Conceio.

image

In this case, for test purposes, I am able to fix the row data by updating the method Filament\Actions\Imports\Jobs\ImportCsv::utf8Encode from

if (is_string($value)) {
     return mb_convert_encoding($value, 'UTF-8', 'UTF-8');
}

to

if (is_string($value)) {
     return mb_convert_encoding($value, 'UTF-8', 'ISO-8859-1');
}

image

But the title column remains with the issue and I am not able to import.

Other than that, I converted my test file to UTF-8 (iconv -f ISO-8859-1 -t utf-8 input-ISO-8859-1.csv -o input-utf8.csv) and it works as expected.

Expected behavior

I expect that the Filament handles CSV files that are not encoded with UTF-8.

Steps to reproduce

  1. Create an import action
  2. Update the getColumns method and add the method guess()
  3. Upload a CSV file encoded with ISO-8859-1 (there is an example in the reproduction repository called input-ISO-8859-1.csv in the project's root). At least one title should have a character like ç, í and ã.
  4. Try to import

Reproduction repository

https://github.com/Rahmon/filamentphp-issues

Relevant log output

Unable to encode attribute [data] for model [Filament\Actions\Imports\Models\FailedImportRow] to JSON: Malformed UTF-8 characters, possibly incorrectly encoded. {"userId":1,"exception":"[object] (Illuminate\\Database\\Eloquent\\JsonEncodingException(code: 0): Unable to encode attribute [data] for model [Filament\\Actions\\Imports\\Models\\FailedImportRow] to JSON: Malformed UTF-8 characters, possibly incorrectly encoded. at /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Database/Eloquent/JsonEncodingException.php:47)
[stacktrace]
#0 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Concerns/HasAttributes.php(1292): Illuminate\\Database\\Eloquent\\JsonEncodingException::forAttribute()
#1 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Concerns/HasAttributes.php(1024): Illuminate\\Database\\Eloquent\\Model->castAttributeAsJson()
#2 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Model.php(2243): Illuminate\\Database\\Eloquent\\Model->setAttribute()
#3 /tmp/filament-issue/vendor/filament/actions/src/Imports/Jobs/ImportCsv.php(133): Illuminate\\Database\\Eloquent\\Model->__set()
#4 /tmp/filament-issue/vendor/filament/actions/src/Imports/Jobs/ImportCsv.php(86): Filament\\Actions\\Imports\\Jobs\\ImportCsv->logFailedRow()
#5 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Filament\\Actions\\Imports\\Jobs\\ImportCsv->handle()
#6 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Container/Util.php(41): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#7 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\\Container\\Util::unwrapIfClosure()
#8 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\\Container\\BoundMethod::callBoundMethod()
#9 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Container/Container.php(662): Illuminate\\Container\\BoundMethod::call()
#10 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(128): Illuminate\\Container\\Container->call()
#11 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(144): Illuminate\\Bus\\Dispatcher->Illuminate\\Bus\\{closure}()
#12 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(119): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#13 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(132): Illuminate\\Pipeline\\Pipeline->then()
#14 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(123): Illuminate\\Bus\\Dispatcher->dispatchNow()
#15 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(144): Illuminate\\Queue\\CallQueuedHandler->Illuminate\\Queue\\{closure}()
#16 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Queue/Middleware/WithoutOverlapping.php(78): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#17 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(183): Illuminate\\Queue\\Middleware\\WithoutOverlapping->handle()
#18 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(119): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#19 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(122): Illuminate\\Pipeline\\Pipeline->then()
#20 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(70): Illuminate\\Queue\\CallQueuedHandler->dispatchThroughMiddleware()
#21 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(102): Illuminate\\Queue\\CallQueuedHandler->call()
#22 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(439): Illuminate\\Queue\\Jobs\\Job->fire()
#23 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(389): Illuminate\\Queue\\Worker->process()
#24 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(333): Illuminate\\Queue\\Worker->runJob()
#25 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(139): Illuminate\\Queue\\Worker->runNextJob()
#26 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(122): Illuminate\\Queue\\Console\\WorkCommand->runWorker()
#27 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Illuminate\\Queue\\Console\\WorkCommand->handle()
#28 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Container/Util.php(41): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#29 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\\Container\\Util::unwrapIfClosure()
#30 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\\Container\\BoundMethod::callBoundMethod()
#31 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Container/Container.php(662): Illuminate\\Container\\BoundMethod::call()
#32 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Console/Command.php(212): Illuminate\\Container\\Container->call()
#33 /tmp/filament-issue/vendor/symfony/console/Command/Command.php(279): Illuminate\\Console\\Command->execute()
#34 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Console/Command.php(181): Symfony\\Component\\Console\\Command\\Command->run()
#35 /tmp/filament-issue/vendor/symfony/console/Application.php(1049): Illuminate\\Console\\Command->run()
#36 /tmp/filament-issue/vendor/symfony/console/Application.php(318): Symfony\\Component\\Console\\Application->doRunCommand()
#37 /tmp/filament-issue/vendor/symfony/console/Application.php(169): Symfony\\Component\\Console\\Application->doRun()
#38 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(196): Symfony\\Component\\Console\\Application->run()
#39 /tmp/filament-issue/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(1183): Illuminate\\Foundation\\Console\\Kernel->handle()
#40 /tmp/filament-issue/artisan(13): Illuminate\\Foundation\\Application->handleCommand()
#41 {main}
danharrin commented 7 months ago

I would appreciate it if you investigated this issue and opened a PR yourself with your findings if possible.

valentin-morice commented 7 months ago

Adding a stream filter in the CanImportRecords trait, getUploadedFileStream method:

use League\Csv\CharsetConverter;

public function getUploadedFileStream(TemporaryUploadedFile $file) {

        CharsetConverter::register();

        $filePath = $file->getRealPath();

        if (config('filament.default_filesystem_disk') !== 's3') {
            $resource = fopen($filePath, mode: 'r');

            $filter = stream_filter_append(
                $resource,
                CharsetConverter::getFiltername('iso-8859-15', 'utf-8'),
                STREAM_FILTER_READ
            );

            return $resource;
        }

        // ...
}

makes the import work:

image

However, the input encoding would have to be set dynamically. This function doesn't seem to work, as of PHP8.1, to automatically detect encoding. Maybe a dropdown with possible encodings could be presented to the user instead?

Rahmon commented 7 months ago

Hi @valentin-morice

I tested your approach and it worked as expected.

Maybe instead of a dropdown with possible encodings presented to the user, it could be an option in the Importer class with the expected charsets and using the function mb_detect_encoding to detect the most likely character encoding.