search

teslamate-org / teslamate

A self-hosted data logger for your Tesla 🚘
https://docs.teslamate.org
MIT License
5.96k stars 741 forks source link

Import from TeslaFi Does Not Appear to do Anything? #3607

Closed keithdw1972 closed 9 months ago

keithdw1972 commented 10 months ago

Is there an existing issue for this?

What happened?

Downloaded a years worth of TeslaFi csvs. Copied them to the docker container, in a 'temp folder. Stopped Teslamate container Copied the files into the ./import directory. Started the Teslamate container.

Expected Behavior

The documentation says to navigate to the /admin page of Teslamate and there will be an import screen, detailing the files.

Steps To Reproduce

Download TeslaFi csvs. Copied to the docker container, in a 'temp folder. Stopped Teslamate container Copied the files into the ./import directory. Start the Teslamate container.

Relevant log output

2, soc: 81, elevation: 25, est_heading: 41, est_lat: 56.051043, est_lng: -4.532245, power: 2, shift_state: "R", range: 211, est_range: 166, heading: 221}
teslamate-1  | 2024-01-11 16:17:23.115 car_id=1 [info] Driving / Start
teslamate-1  | 2024-01-11 16:27:22.411 [info] GET https://nominatim.openstreetmap.org/reverse -> 200 (180.853 ms)
teslamate-1  | 2024-01-11 16:27:22.903 [info] GET https://nominatim.openstreetmap.org/reverse -> 200 (488.411 ms)
teslamate-1  | 2024-01-11 16:27:22.917 [info] End of drive initiated by: %TeslaApi.Vehicle.State.Drive{gps_as_of: 1704990440, heading: 84, latitude: 56.063771, longitude: -4.450848, native_latitude: 56.063771, native_location_supported: 1, native_longitude: -4.450848, native_type: "wgs", power: 2, shift_state: "P", speed: 0, timestamp: 1704990442149}
teslamate-1  | 2024-01-11 16:27:22.917 car_id=1 [info] Driving / Ended / 7 km – 10 min
teslamate-1  | 2024-01-11 16:27:22.917 car_id=1 [info] Start / :online
teslamate-1  | 2024-01-11 16:28:57.911 car_id=1 [info] Received stream data: :inactive
teslamate-1  | 2024-01-11 16:30:39.430 [info] GET https://owner-api.teslamotors.com/api/1/vehicles/929640529449945/vehicle_data -> 408 (73.107 ms)
teslamate-1  | 2024-01-11 16:30:49.583 car_id=1 [info] Start / :asleep
teslamate-1  | 2024-01-11 16:30:49.589 car_id=1 [info] Disconnecting ...
teslamate-1  | 2024-01-11 16:57:27.989 [info] GET /geo-fences/new
teslamate-1  | 2024-01-11 16:57:27.994 [info] Sent 200 in 4ms
teslamate-1  | 2024-01-11 16:58:09.468 [info] GET /geo-fences/new
teslamate-1  | 2024-01-11 16:58:09.473 [info] Sent 200 in 4ms
teslamate-1  | 2024-01-11 17:00:58.941 [notice] SIGTERM received - shutting down
teslamate-1  | 
teslamate-1  | 2024-01-11 17:01:44.159 [info] Migrations already up
teslamate-1  | 2024-01-11 17:01:46.897 [info] System Info: Erlang/OTP 26 (jit)
teslamate-1  | 2024-01-11 17:01:46.897 [info] Version: 1.28.2
teslamate-1  | 2024-01-11 17:01:47.746 [info] POST https://auth.tesla.com/oauth2/v3/token -> 200 (807.845 ms)
teslamate-1  | 2024-01-11 17:01:47.746 [info] Refreshed api tokens
teslamate-1  | 2024-01-11 17:01:47.752 [info] Scheduling token refresh in 6 h
teslamate-1  | 2024-01-11 17:01:47.754 [info] Running TeslaMateWeb.Endpoint with cowboy 2.10.0 at :::4000 (http)
teslamate-1  | 2024-01-11 17:01:47.756 [info] Access TeslaMateWeb.Endpoint at http://localhost
teslamate-1  | 2024-01-11 17:01:47.909 [info] Starting logger for 'Williams'
teslamate-1  | 2024-01-11 17:01:47.920 [info] MQTT connection has been established
teslamate-1  | 2024-01-11 17:01:47.992 [info] GET /
teslamate-1  | 2024-01-11 17:01:48.003 car_id=1 [info] Start / :asleep
teslamate-1  | 2024-01-11 17:01:48.011 [info] Sent 200 in 18ms
teslamate-1  | 2024-01-11 17:01:48.103 [info] GET /geo-fences
teslamate-1  | 2024-01-11 17:01:48.107 [info] Sent 200 in 4ms
teslamate-1  | 2024-01-11 17:01:53.009 [info] GET /geo-fences
teslamate-1  | 2024-01-11 17:01:53.013 [info] Sent 200 in 3ms

Screenshots

No response

Additional data

No response

Type of installation

Docker

Version

1.28.2

dyxyl commented 9 months ago

You say

Copied them to the docker container, in a 'temp folder.

and then

Copied the files into the ./import directory.

but the instructions say to "Copy the exported CSV files into a directory named import next to the docker-compose.yml". That is on the docker host, not in the docker container. It gets mounted in the container by docker.

If that's not enough to solve it, then the output of:

docker exec teslamate-1 ls -la import

would be useful (replace teslamate-1 with the name of the TeslaMate container).

damccull commented 9 months ago

It's not explicitly called out, which is annoying, but you look at the filenames in the example in the docs and see they're all prefixed with "TeslaFi". If you rename your files by adding that prefix, but leave the rest of the filename as is, it should work.

However...you will likely need to edit every single line in your csv files to change the 5th field, "car id" I think, to a value of 1 to get them to import properly.

So, two tasks to import: 1) Rename all files to add TeslaFi as a prefix 2) Add car id of 1 to field 5 in all rows of all csvs. Note: DO NOT let excel modify the date format of the date fields or it'll also fail. Excel will do this automatically. I do not know how to prevent it. I used a text editor and regex to handle my car id field.

keithdw1972 commented 9 months ago

Thanks damccull, it now recognises that there are files to import. The import fails with an invalid date format now though:

2024-01-16 10:06:46.647 [error] Task #PID<0.2591.0> started from TeslaMate.Import terminating (MatchError) no match of right hand side value: {:error, {:invalid_date_format, "30/12/2022 22:32"}} (teslamate 1.28.2) lib/teslamate/import/line_parser.ex:80: TeslaMate.Import.LineParser.into_vehicle/3 (stdlib 5.1.1) maps.erl:416: :maps.fold_1/4 (teslamate 1.28.2) lib/teslamate/import/line_parser.ex:20: TeslaMate.Import.LineParser.parse/2 (elixir 1.15.7) lib/task/supervised.ex:101: Task.Supervised.invoke_mfa/2 (elixir 1.15.7) lib/task/supervised.ex:36: Task.Supervised.reply/4 Function: &:erlang.apply/2 Args: [#Function<7.111590375/1 in TeslaMate.Import.create_event_streams/2>, [%{"charge_energy_added" => "", "outside_temp" => "", "elevation" => "", "rear_seat_heaters" => "", "fan_status" => "", "scheduled_charging_start_time" => "", "spoiler_type" => "", "id" => "", "max_range_charge_counter" => "", "locked" => "", "gui_temperature_units" => "", "battery_range" => "", "vin" => "", "idleNumber" => "0", "calendar_enabled" => "", "rhd" => "", "rear_seat_type" => "", "heading" => "", "seat_heater_right" => "", "charger_phases" => "", "outside_tempF" => "", "rerunning" => "0", "trip_charging" => "", "fast_charger_present" => "", "car_type" => "", "calendar_supported" => "", "gui_distance_units" => "", "driver_temp_setting" => "", "gps_as_of" => "", "right_temp_direction" => "", "notifications_supported" => "", "sun_roof_installed" => "", "api_version" => "", "seat_heater_rear_right_back" => "", "charging_state" => "", "gui_range_display" => "", "remote_start_enabled" => "", "seat_heater_rear_left_back" => "", "ft" => "", "eu_vehicle" => "", "df" => "", "charge_current_request_max" => "", "is_front_defroster_on" => "", "gui_charge_rate_units" => "", "valet_mode" => "", "pf" => "", "running" => "0", ...}]] 2024-01-16 10:06:46.647 [error] Task #PID<0.2590.0> started from TeslaMate.Import terminating (MatchError) no match of right hand side value: {:error, {:invalid_date_format, "30/12/2022 22:21"}} (teslamate 1.28.2) lib/teslamate/import/line_parser.ex:80: TeslaMate.Import.LineParser.into_vehicle/3 (stdlib 5.1.1) maps.erl:416: :maps.fold_1/4 (teslamate 1.28.2) lib/teslamate/import/line_parser.ex:20: TeslaMate.Import.LineParser.parse/2 (elixir 1.15.7) lib/task/supervised.ex:101: Task.Supervised.invoke_mfa/2 (elixir 1.15.7) lib/task/supervised.ex:36: Task.Supervised.reply/4 Function: &:erlang.apply/2 Args: [#Function<7.111590375/1 in TeslaMate.Import.create_event_streams/2>, [%{"charge_energy_added" => "", "outside_temp" => "", "elevation" => "", "rear_seat_heaters" => "", "fan_status" => "", "scheduled_charging_start_time" => "", "spoiler_type" => "", "id" => "", "max_range_charge_counter" => "", "locked" => "", "gui_temperature_units" => "", "battery_range" => "", "vin" => "", "idleNumber" => "0", "calendar_enabled" => "", "rhd" => "", "rear_seat_type" => "", "heading" => "", "seat_heater_right" => "", "charger_phases" => "", "outside_tempF" => "", "rerunning" => "0", "trip_charging" => "", "fast_charger_present" => "", "car_type" => "", "calendar_supported" => "", "gui_distance_units" => "", "driver_temp_setting" => "", "gps_as_of" => "", "right_temp_direction" => "", "notifications_supported" => "", "sun_roof_installed" => "", "api_version" => "", "seat_heater_rear_right_back" => "", "charging_state" => "", "gui_range_display" => "", "remote_start_enabled" => "", "seat_heater_rear_left_back" => "", "ft" => "", "eu_vehicle" => "", "df" => "", "charge_current_request_max" => "", "is_front_defroster_on" => "", "gui_charge_rate_units" => "", "valet_mode" => "", "pf" => "", "running" => "0", ...}]] 2024-01-16 10:06:46.651 [error] GenStateMachine TeslaMate.Import terminating ** (FunctionClauseError) no function clause matching in anonymous fn/1 in TeslaMate.Import.create_event_streams/2 (teslamate 1.28.2) anonymous fn({:exit, {{:badmatch, {:error, {:invalid_date_format, "30/12/2022 22:21"}}}, [{TeslaMate.Import.LineParser, :into_vehicle, 3, [file: ~c"lib/teslamate/import/line_parser.ex", line: 80]}, {:maps, :fold_1, 4, [file: ~c"maps.erl", line: 416]}, {TeslaMate.Import.LineParser, :parse, 2, [file: ~c"lib/teslamate/import/line_parser.ex", line: 20]}, {Task.Supervised, :invoke_mfa, 2, [file: ~c"lib/task/supervised.ex", line: 101]}, {Task.Supervised, :reply, 4, [file: ~c"lib/task/supervised.ex", line: 36]}]}}) in TeslaMate.Import.create_event_streams/2

damccull commented 9 months ago

That's almost guaranteed to be because you opened it in Excel to add the 1 to the car id column and saved it. Excel saves dates in it's own format which is not going to match what the files originally had. That's why I mentioned that I used a regex find and replace in a text editor to do mine. Explaining that is beyond my available time though... You will need to find a way to edit the csv file without using Excel, or a way to tell excel to leave the existing date columns intact.

keithdw1972 commented 9 months ago

thats kinda what i expected you to say... ive redownloaded and checked the format, all fields 'appear' to be in the same format...

then again, i am using Excel for both (shrug)

damccull commented 9 months ago

thats kinda what i expected you to say... ive redownloaded and checked the format, all fields 'appear' to be in the same format...

then again, i am using Excel for both (shrug)

Open the raw csv in notepad or, better, vs code or another fancy editor instead of Excel. Excel will display the pretty date in the cell you see, but underneath converts it to an integer. It is likely the number of seconds since a specific time defined by Microsoft. Anyway, open a csv saved by Excel and the raw one from teslafi and look at the differences in the dates.

You can probably use any of the tons of free online or installed csv editors out there to do it though. Just Google "csv editor" and add "online" or your operating system.

It would be nice to see the importer updated to recognize the missing car id and sub in a default. If I knew anything about elixir I'd consider opening a pr, but I do not.

keithdw1972 commented 9 months ago

Miller is your friend here... although that involved remembering how to drive it!

All sorted... thanks for the help.