search

Infrasurvey / iot-portal-backend

0 stars 0 forks source link

Different bugs when fetching data from FTP server #3

Open guillaumeVolery opened 2 years ago

guillaumeVolery commented 2 years ago

Problem description:

The command php artisan geomon:fetch_ftp is stopped with different kind of reported errors.

Expected behavior:

The command shall run until the end of fetching without problems.

guillaumeVolery commented 2 years ago

Seen error:

Processing Base Station 35

   Illuminate\Database\QueryException 

  SQLSTATE[01000]: Warning: 1265 Data truncated for column 'reference_altitude' at row 1 (SQL: insert into `configuration_base_stations` (`device_base_station_id`, `file_id`, 
`continuous_mode`, `reset`, `wakeup_period_in_minutes`, `session_duration_in_minutes`, `reference_gps_module`, `reference_latitude`, `reference_longitude`, `reference_altitude`, `non_continuous_store_binr_to_ftp`, `updated_at`, `created_at`) values (31, 38409, 0, 1, 7, 1, 10, 49.3102579, 2.7266187, 41.5.000, 1, 2022-03-30 12:15:25, 2022-03-30 12:15:25))

  at vendor/laravel/framework/src/Illuminate/Database/Connection.php:685
    681▕         // If an exception occurs when attempting to run a query, we'll format the error
    682▕         // message to include the bindings with SQL, which will make this exception a
    683▕         // lot more helpful to the developer instead of just the database's errors.
    684▕         catch (Exception $e) {
  ➜ 685▕             throw new QueryException(
    686▕                 $query, $this->prepareBindings($bindings), $e
    687▕             );
    688▕         }
    689▕ 

      +12 vendor frames
  13  app/Console/Commands/FetchDeviceData.php:312
      Illuminate\Database\Eloquent\Model::save()

  14  app/Console/Commands/FetchDeviceDataFtp.php:138
      App\Console\Commands\FetchDeviceData::fetch()
guillaumeVolery commented 2 years ago

Station 35 - Bug identification:

The following configuration files

contained all the same typo error on the ALTITUDE parameter:

ALTITUDE=41.5.000 ; Meters

Note: 2 points in the decimal same number.

Bug resolution:

File corrected and ALTITUDE value set to 415.000 meters.

Verification procedure:

Re-run the php artisan geomon:fetch_ftp command and check that all stations are correctly processed.

Conclusion:

Bug resolved but not sufficient. File must be kept as corrupted and fetching function shall set the invalid column correctly.

guillaumeVolery commented 2 years ago

New error found in base station 32

Processing measure : GM_BASE_0032/200929_20

   ErrorException 

  Trying to get property 'id' of non-object

  at app/Console/Commands/FetchDeviceData.php:703
    699▕                                                 })
    700▕                                                 ->where('device_base_station_id', $deviceBaseStation->id)
    701▕                                                 ->first();
    702▕                     
  ➜ 703▕                     $position = Position::where([['device_rover_id', $deviceRover->id], ['file_id', $file->id]])->first();
    704▕                     if ($position == null)
    705▕                     {
    706▕                         $position = new Position;
    707▕                         $position->device_rover_id = $deviceRover->id;

  1   app/Console/Commands/FetchDeviceData.php:703
      Illuminate\Foundation\Bootstrap\HandleExceptions::handleError("Trying to get property 'id' of non-object", "/srv/www/app/Console/Commands/FetchDeviceData.php")

  2   app/Console/Commands/FetchDeviceDataFtp.php:138
      App\Console\Commands\FetchDeviceData::fetch()
guillaumeVolery commented 2 years ago

New error found:

Processing measure : GM_BASE_0002/160511_19

   ErrorException 

  sort() expects parameter 1 to be array, bool given

  at app/Console/Commands/FetchDeviceDataFtp.php:83
     79▕     protected function listDir($dirPath)
     80▕     {
     81▕         $dirPath = "/data/Geomon/" . $dirPath;
     82▕         $paths = ftp_nlist($this->ftp, $dirPath);
  ➜  83▕         sort($paths);
     84▕ 
     85▕         // Remove "/data/Geomon/" of each path
     86▕         foreach($paths as $key => $path)
     87▕         {

  1   app/Console/Commands/FetchDeviceDataFtp.php:83
      sort()

  2   app/Console/Commands/FetchDeviceData.php:406
      App\Console\Commands\FetchDeviceDataFtp::listDir("/data/Geomon/GM_BASE_0002/160511_19")
guillaumeVolery commented 2 years ago

All error resolved in commit a19c687.

Fetching all base stations is not blocking anymore.

Resolved.

Base station GM_BASE_0035:

id;device_base_station_id;file_id;continuous_mode;reset;wakeup_period_in_minutes;session_start_time;session_period_in_wakeup_period;session_duration_in_minutes;non_continuous_store_binr_to_ftp;reference_gps_module;reference_latitude;reference_longitude;reference_altitude;validity;created_at;updated_at
354;38;41345;0;1;7;\N;\N;1;1;10;49.3102579;2.7266187;0;corrupted;2022-04-12 14:23:53;2022-04-12 14:23:53
377;38;41368;0;1;7;\N;\N;1;1;10;49.310278;2.726607;0;corrupted;2022-04-12 14:23:55;2022-04-12 14:23:55
378;38;41369;0;1;1440;\N;\N;10;1;10;49.310278;2.726607;0;corrupted;2022-04-12 14:23:55;2022-04-12 14:23:55
379;38;41370;0;1;7;\N;\N;1;1;10;49.3102579;2.7266187;0;corrupted;2022-04-12 14:23:55;2022-04-12 14:23:55
380;38;41371;0;1;360;\N;\N;10;1;10;49.3102579;2.7266187;0;corrupted;2022-04-12 14:23:55;2022-04-12 14:23:55
381;38;41372;0;1;1440;\N;\N;30;1;10;49.3102579;2.7266187;0;corrupted;2022-04-12 14:23:55;2022-04-12 14:23:55

Base station GM_BASE_0032:

Base station GM_BASE_0002: