search

teslamate-org / teslamate

A self-hosted data logger for your Tesla 🚘
https://docs.teslamate.org
MIT License
5.96k stars 741 forks source link

TeslaFi data import failure #3158

Closed sirismile closed 11 months ago

sirismile commented 1 year ago

Is there an existing issue for this?

What happened?

not able to import the data after following the steps in the documentation

Expected Behavior

not able to import the data after follow the steps in the documentation the log says data incomplete, but dont know where to debug.

apologies, im new to linux and ubuntu.

my export file from teslafi does indicate a lot of empty spaces. but dont know if its intentionally done like that?! i looked one of the previous issue that uploaded file, and mine looks much simpler.

Steps To Reproduce

No response

Relevant log output

teslamate_1  | 2023-03-24 01:15:35.695 [info] Importing 1 file(s) ...
teslamate_1  | 2023-03-24 01:15:44.540 [error] GenStateMachine TeslaMate.Import terminating
teslamate_1  | ** (RuntimeError) vehicle data is incomplete
teslamate_1  |     (teslamate 1.27.2) lib/teslamate/import.ex:274: TeslaMate.Import.create_car/1
teslamate_1  |     (teslamate 1.27.2) lib/teslamate/import.ex:128: TeslaMate.Import.handle_event/4
teslamate_1  |     (stdlib 3.17.2.1) gen_statem.erl:1205: :gen_statem.loop_state_callback/11
teslamate_1  |     (stdlib 3.17.2.1) proc_lib.erl:226: :proc_lib.init_p_do_apply/3

Screenshots

not able to import the data after follow the steps in the documentation the log says data incomplete, but dont know where to debug.

apologies, im new to linux and ubuntu.

my export file from teslafi does indicate a lot of empty spaces. but dont know if its intentionally done like that?!

Additional data

No response

Type of installation

Docker

Version

3

MagoKimbra commented 1 year ago

Same problem, I tried several times today, import I follow all the guide, but nothing imported data...

Zjaaspoer commented 1 year ago

Agree, tried several exports & imports from TeslaFi into TeslaMate, same error every time. Running the latest docker of TeslaMate

Kjehlje commented 1 year ago

+1, same problem here. Same log output.

Etus12 commented 1 year ago

+1, same problem here. Same log output.

augustcove commented 1 year ago

+1, same problem and log output.

As an update, I compared the columns for my files exported from TeslaFi with the unit test files and the format appear to be the same. I also tried removing any rows in which the vin or display_name was empty.

vzhivkov commented 1 year ago

+1 same problem happened to me yesterday.

ironfistperez commented 1 year ago

Same issue here. Any idea if this will ever get fixed? Got months of data in Teslafi that would be nice to move over.

2023-06-21` 17:08:55.431 [info] Importing 1 file(s) ... 2023-06-21 17:09:01.353 [error] GenStateMachine TeslaMate.Import terminating ** (RuntimeError) vehicle data is incomplete (teslamate 1.27.3) lib/teslamate/import.ex:274: TeslaMate.Import.create_car/1 (teslamate 1.27.3) lib/teslamate/import.ex:128: TeslaMate.Import.handle_event/4 (stdlib 5.0.1) gen_statem.erl:1379: :gen_statem.loop_state_callback/11 (stdlib 5.0.1) proc_lib.erl:241: :proc_lib.init_p_do_apply/3

kmccb commented 1 year ago

Anyone get this fixed? I can't import my teslafi data either :(

vzhivkov commented 1 year ago

Here is what I did to make it work. The downloaded files from TeslaFi I edited using this website https://www.convertcsv.com/csv-viewer-editor.htm?utm_content=cmp-true - I changed values in 2 columns. I entered value "1" in all rows in the column vehicle_id . Also my car didn't have a name. I entered a name in my Tesla app and entered the same for all rows in the column display_name in the file from TeslaFi. Then the import in Teslamate went fine.

kmccb commented 1 year ago

Here is what I did to make it work. The downloaded files from TeslaFi I edited using this website https://www.convertcsv.com/csv-viewer-editor.htm?utm_content=cmp-true - I changed values in 2 columns. I entered value "1" in all rows in the column vehicle_id . Also my car didn't have a name. I entered a name in my Tesla app and entered the same for all rows in the column display_name in the file from TeslaFi. Then the import in Teslamate went fine.

Thanks for replying. I tried doing that and it still fails.. I didn't have anything in the vehicle_id field so I entered 1 in all them just as you did.. I had some blanks and then two different names in my display_name so I changed them all to match what's in my app now.. Tried to import and no bueno.. Gives an error about invalid date format..

2023-06-23 14:18:22 stdout 2023-06-23 14:18:22.177 [notice] SIGTERM received - shutting down 2023-06-23 14:11:35 stdout (stdlib 5.0.1) gen_statem.erl:1379: :gen_statem.loop_state_callback/11 2023-06-23 14:11:35 stdout (teslamate 1.27.3) lib/teslamate/import.ex:128: TeslaMate.Import.handle_event/4 2023-06-23 14:11:35 stdout (teslamate 1.27.3) lib/teslamate/import.ex:280: TeslaMate.Import.create_car/1 2023-06-23 14:11:35 stdout (elixir 1.15.0-rc.1) lib/enum.ex:1159: Enum.find/3 2023-06-23 14:11:35 stdout (elixir 1.15.0-rc.1) lib/stream.ex:1828: Enumerable.Stream.do_each/4 2023-06-23 14:11:35 stdout (elixir 1.15.0-rc.1) lib/task/supervised.ex:386: Task.Supervised.stream_deliver/7 2023-06-23 14:11:35 stdout (elixir 1.15.0-rc.1) lib/stream.ex:613: anonymous fn/4 in Stream.map/2 2023-06-23 14:11:35 stdout (teslamate 1.27.3) anonymous fn({:exit, {{:badmatch, {:error, {:invalid_date_format, "6/9/2023 8:34"}}}, [{TeslaMate.Import.LineParser, :into_vehicle, 3, [file: ~c"lib/teslamate/import/line_parser.ex", line: 80]}, {:maps, :fold_1, 4, [file: ~c"maps.erl", line: 416]}, {TeslaMate.Import.LineParser, :parse, 2, [file: ~c"lib/teslamate/import/line_parser.ex", line: 20]}, {Task.Supervised, :invoke_mfa, 2, [file: ~c"lib/task/supervised.ex", line: 101]}, {Task.Supervised, :reply, 4, [file: ~c"lib/task/supervised.ex", line: 36]}]}}) in TeslaMate.Import.create_event_streams/2 2023-06-23 14:11:35 stdout ** (FunctionClauseError) no function clause matching in anonymous fn/1 in TeslaMate.Import.create_event_streams/2 2023-06-23 14:11:35 stdout 2023-06-23 14:11:35.397 [error] GenStateMachine TeslaMate.Import terminating 2023-06-23 14:11:35 stdout Args: [#Function<7.111590375/1 in TeslaMate.Import.create_event_streams/2>, [%{"charge_energy_added" => "", "outside_temp" => "", "elevation" => "", "rear_seat_heaters" => "", "fan_status" => "", "scheduled_charging_start_time" => "", "spoiler_type" => "", "id" => "", "max_range_charge_counter" => "", "locked" => "", "gui_temperature_units" => "", "battery_range" => "", "vin" => "", "idleNumber" => "0", "calendar_enabled" => "", "rhd" => "", "rear_seat_type" => "", "heading" => "", "seat_heater_right" => "", "charger_phases" => "", "outside_tempF" => "", "rerunning" => "0", "trip_charging" => "", "fast_charger_present" => "", "car_type" => "", "calendar_supported" => "", "gui_distance_units" => "", "driver_temp_setting" => "", "gps_as_of" => "", "right_temp_direction" => "", "notifications_supported" => "", "sun_roof_installed" => "", "api_version" => "", "seat_heater_rear_right_back" => "", "charging_state" => "", "gui_range_display" => "", "remote_start_enabled" => "", "seat_heater_rear_left_back" => "", "ft" => "", "eu_vehicle" => "", "df" => "", "charge_current_request_max" => "", "is_front_defroster_on" => "", "gui_charge_rate_units" => "", "valet_mode" => "", "pf" => "", "running" => "0", ...}]] 2023-06-23 14:11:35 stdout Function: &:erlang.apply/2 2023-06-23 14:11:35 stdout (elixir 1.15.0-rc.1) lib/task/supervised.ex:36: Task.Supervised.reply/4 2023-06-23 14:11:35 stdout (elixir 1.15.0-rc.1) lib/task/supervised.ex:101: Task.Supervised.invoke_mfa/2 2023-06-23 14:11:35 stdout (teslamate 1.27.3) lib/teslamate/import/line_parser.ex:20: TeslaMate.Import.LineParser.parse/2 2023-06-23 14:11:35 stdout (stdlib 5.0.1) maps.erl:416: :maps.fold_1/4 2023-06-23 14:11:35 stdout (teslamate 1.27.3) lib/teslamate/import/line_parser.ex:80: TeslaMate.Import.LineParser.into_vehicle/3 2023-06-23 14:11:35 stdout ** (MatchError) no match of right hand side value: {:error, {:invalid_date_format, "6/9/2023 8:34"}} 2023-06-23 14:11:35 stdout 2023-06-23 14:11:35.390 [error] Task #PID<0.2594.0> started from TeslaMate.Import terminating 2023-06-23 14:11:35 stdout Args: [#Function<7.111590375/1 in TeslaMate.Import.create_event_streams/2>, [%{"charge_energy_added" => "16.88", "outside_temp" => "19.5", "elevation" => "", "rear_seat_heaters" => "", "fan_status" => "0", "scheduled_charging_start_time" => "", "spoiler_type" => "", "id" => "", "max_range_charge_counter" => "", "locked" => "1", "gui_temperature_units" => "", "battery_range" => "192.1", "vin" => "5YJ3E1EB4NF264444", "idleNumber" => "1", "calendar_enabled" => "", "rhd" => "0", "rear_seat_type" => "0", "heading" => "193", "seat_heater_right" => "0", "charger_phases" => "1", "outside_tempF" => "67", "rerunning" => "0", "trip_charging" => "", "fast_charger_present" => "0", "car_type" => "", "calendar_supported" => "", "gui_distance_units" => "", "driver_temp_setting" => "22", "gps_as_of" => "", "right_temp_direction" => "", "notifications_supported" => "", "sun_roof_installed" => "", "api_version" => "", "seat_heater_rear_right_back" => "", "charging_state" => "Charging", "gui_range_display" => "", "remote_start_enabled" => "", "seat_heater_rear_left_back" => "", "ft" => "0", "eu_vehicle" => "0", "df" => "0", "charge_current_request_max" => "12", "is_front_defroster_on" => "0", "gui_charge_rate_units" => "", "valet_mode" => "0", "pf" => "0", "running" => "0", ...}]] 2023-06-23 14:11:35 stdout Function: &:erlang.apply/2 2023-06-23 14:11:35 stdout (elixir 1.15.0-rc.1) lib/task/supervised.ex:36: Task.Supervised.reply/4 2023-06-23 14:11:35 stdout (elixir 1.15.0-rc.1) lib/task/supervised.ex:101: Task.Supervised.invoke_mfa/2 2023-06-23 14:11:35 stdout (teslamate 1.27.3) lib/teslamate/import/line_parser.ex:20: TeslaMate.Import.LineParser.parse/2 2023-06-23 14:11:35 stdout (stdlib 5.0.1) maps.erl:416: :maps.fold_1/4 2023-06-23 14:11:35 stdout (teslamate 1.27.3) lib/teslamate/import/line_parser.ex:80: TeslaMate.Import.LineParser.into_vehicle/3 2023-06-23 14:11:35 stdout ** (MatchError) no match of right hand side value: {:error, {:invalid_date_format, "6/9/2023 8:56"}} 2023-06-23 14:11:35 stdout 2023-06-23 14:11:35.391 [error] Task #PID<0.2595.0> started from TeslaMate.Import terminating 2023-06-23 14:11:35 stdout Args: [#Function<7.111590375/1 in TeslaMate.Import.create_event_streams/2>, [%{"charge_energy_added" => "16.9", "outside_temp" => "19.5", "elevation" => "", "rear_seat_heaters" => "", "fan_status" => "0", "scheduled_charging_start_time" => "", "spoiler_type" => "", "id" => "", "max_range_charge_counter" => "", "locked" => "1", "gui_temperature_units" => "", "battery_range" => "192.1", "vin" => "5YJ3E1EB4NF264444", "idleNumber" => "0", "calendar_enabled" => "", "rhd" => "0", "rear_seat_type" => "0", "heading" => "193", "seat_heater_right" => "0", "charger_phases" => "1", "outside_tempF" => "67", "rerunning" => "0", "trip_charging" => "", "fast_charger_present" => "0", "car_type" => "", "calendar_supported" => "", "gui_distance_units" => "", "driver_temp_setting" => "22", "gps_as_of" => "", "right_temp_direction" => "", "notifications_supported" => "", "sun_roof_installed" => "", "api_version" => "", "seat_heater_rear_right_back" => "", "charging_state" => "Charging", "gui_range_display" => "", "remote_start_enabled" => "", "seat_heater_rear_left_back" => "", "ft" => "0", "eu_vehicle" => "0", "df" => "0", "charge_current_request_max" => "12", "is_front_defroster_on" => "0", "gui_charge_rate_units" => "", "valet_mode" => "0", "pf" => "0", "running" => "0", ...}]] 2023-06-23 14:11:35 stdout Function: &:erlang.apply/2 2023-06-23 14:11:35 stdout (elixir 1.15.0-rc.1) lib/task/supervised.ex:36: Task.Supervised.reply/4 2023-06-23 14:11:35 stdout (elixir 1.15.0-rc.1) lib/task/supervised.ex:101: Task.Supervised.invoke_mfa/2 2023-06-23 14:11:35 stdout (teslamate 1.27.3) lib/teslamate/import/line_parser.ex:20: TeslaMate.Import.LineParser.parse/2 2023-06-23 14:11:35 stdout (stdlib 5.0.1) maps.erl:416: :maps.fold_1/4 2023-06-23 14:11:35 stdout (teslamate 1.27.3) lib/teslamate/import/line_parser.ex:80: TeslaMate.Import.LineParser.into_vehicle/3 2023-06-23 14:11:35 stdout ** (MatchError) no match of right hand side value: {:error, {:invalid_date_format, "6/9/2023 8:57"}} 2023-06-23 14:11:35 stdout 2023-06-23 14:11:35.391 [error] Task #PID<0.2596.0> started from TeslaMate.Import terminating 2023-06-23 14:11:35 stdout 2023-06-23 14:11:35.367 [info] Importing 1 file(s) ...

ironfistperez commented 1 year ago

Yes, I had the same issue with the date after editing the .csv file. Make sure you change the format to m/d/yyyy hh:mm For me it was only using one digit for the hour and after changing it now I have all my data imported!
By the way, I had to do this in Excel.

vzhivkov commented 1 year ago

You've probably changed the date format when you edited the file. The original file from TeslaFi comes with date format like this: "2023-06-01 00:00:41" and this is what Teslamate expects. Check if the original file from TeslaFi comes in this format viewing the file in Notepad or similar text editor that only shows the content without changing it. Then edit the csv properly without changing the date format.

kmccb commented 1 year ago

Yes, I had the same issue with the date after editing the .csv file. Make sure you change the format to m/d/yyyy hh:mm For me it was only using one digit for the hour and after changing it now I have all my data imported! By the way, I had to do this in Excel.

Thanks.. After doing this it now imports without errors.. However, it's only giving me 5 charges (out of 23 for example).. Any ideas on this?

0xAl3xH commented 1 year ago

same issue

0xAl3xH commented 1 year ago

Actually, after getting the CSV files from #563, it seems to import without any issues. I would suggest running the script form there if anyone has this issue.

dblacknc commented 1 year ago

The #563 script didn't work for me either - UI just comes back as if nothing happened, and logs show:

teslamate_1 | 2023-09-07 15:21:57.232 [info] GET /import teslamate_1 | 2023-09-07 15:21:57.248 [info] Sent 200 in 16ms teslamate_1 | 2023-09-07 15:22:06.086 [info] Importing 2 file(s) ... teslamate_1 | 2023-09-07 15:22:15.683 [error] GenStateMachine TeslaMate.Import terminating teslamate_1 | ** (RuntimeError) vehicle data is incomplete teslamate_1 | (teslamate 1.27.3) lib/teslamate/import.ex:274: TeslaMate.Import.create_car/1 teslamate_1 | (teslamate 1.27.3) lib/teslamate/import.ex:128: TeslaMate.Import.handle_event/4 teslamate_1 | (stdlib 5.0.1) gen_statem.erl:1379: :gen_statem.loop_state_callback/11 teslamate_1 | (stdlib 5.0.1) proc_lib.erl:241: :proc_lib.init_p_do_apply/3

Guess I'll try some manual editing.

dblacknc commented 1 year ago

Setting vehicle_id to 1 and my car's name on all the rows seems to have worked.

brad628 commented 1 year ago

Thanks! I was having this issue. This is what worked for me:

  1. Add TeslaFi to the start of file name
  2. Open in Excel and add a 1 to all rows of the vehicle_id column (likely E)
  3. Re-save as CSV (can also do step 1 here)
github-actions[bot] commented 11 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

erichstark commented 10 months ago

Thanks! I was having this issue. This is what worked for me:

  1. Add TeslaFi to the start of file name
  2. Open in Excel and add a 1 to all rows of the vehicle_id column (likely E)
  3. Re-save as CSV (can also do step 1 here)

After this update I no longer have import error in logs, but the data is still not imported.