Datamanagement Tools

PHP classes and helpers for managing data read from text files

Data\FileReader read text files
Data\CsvFileReader read CSV files
Data\FixedWidthReader reads text files that contain data in fixed width columns
Data\Helper helper functions like SC() that converts from spreadsheet column name to index of array generated by (e.g.) CsvFileReader->getLine()

Examples

Work on CSV data

<?php

require __DIR__ . '/../vendor/autoload.php';

use Fiedsch\Data\File\CsvReader;

try {

  $reader = new CsvReader("testdata.csv", ";");

  // Read and handle all lines containing data.

  while (($line = $reader->getLine()) !== null) {
    // ignore empty lines (i.e. lines containing no data)
    if (!$reader->isEmpty($line)) {
      print_r($line);
    }
  }
  // $reader->close(); // not needed as it will be automatically called when there are no more lines

} catch (Exception $e) {
    print $e->getMessage() . "\n";
}

Features

As of v0.3.2 the typical boilerplate "open file, read every non-empty line, close file" can be written in a fancier way. Use the optional parameter to getLine():

 <?php

   while (($line = $reader->getLine(Reader::SKIP_EMPTY_LINES)) !== null) {
       print_r($line);
   }

Data augmentation

<?php

require __DIR__ . '/../vendor/autoload.php';

use Fiedsch\Data\File\CsvReader;
use Fiedsch\Data\Augmentation\Augmentor;
use Fiedsch\Data\Augmentation\Provider\TokenServiceProvider;
use Fiedsch\Data\File\CsvWriter;

try {

  $augmentor = new Augmentor();

  $augmentor->register(new TokenServiceProvider());

  $augmentor->addRule('token', function (Augmentor $augmentor, $data) {
     return [ 'token' => $augmentor['token']->getUniqueToken() ];
   });

   $reader = new CsvReader("testdata.csv", ";");

   $writer = new CsvWriter("testdata.augmented.txt", "\t");

   $header_written = false;

   while (($line = $reader->getLine(Reader::SKIP_EMPTY_LINES)) !== null) {
     $result = $augmentor->augment($line);
     if (!$header_written) {
        $writer->printLine(array_merge(['input_line'], array_keys($result), $reader->getHeader()));
        $header_written = true;
     }
     $writer->printLine(array_merge([$reader->getLineNumber()], $result, $line));
   }

   $writer->close();

 } catch (Exception $e) {
     print $e->getMessage() . "\n";
 }

Creating Tokens

Method one: let the TokenCreator make sure, we have unique tokens:

 <?php

 require __DIR__ . '/../vendor/autoload.php';

 use Fiedsch\Data\Utility\TokenCreator;
 use Fiedsch\Data\File\Writer;

$creator = new TokenCreator(10, TokenCreator::UPPER);

$output = new Writer('mytokens.txt');
$numTokens = 1000;

while ($numTokens-- > 0) {
  $token = $creator->getUniqueToken();
  $output->printLine([$token]);
}
$output->close();

Method two: generate tokens first and then check if they are unique. This might be faster and less resource consuming for large amounts of tokens:

  // same as above, exept 
  // $token = $creator->getUniqueToken();
  // becomes
  $token = $creator->cretateToken();

Check that the generated tokens are unique

echo " both lines show the same numbers, there were no duplicate tokens"
wc -l mytokens.csv
sort mytokens.csv | uniq | wc -l

fiedsch / datamanagement

readme