search

antonroman / smart_meter_data_analysis

This repository contains all the code developed to analyze the smart meter data with HTM and LSTM
2 stars 0 forks source link

Generate script to build S02 and S05 CSV files from JSON files #1

Closed antonroman closed 3 years ago

antonroman commented 3 years ago

There is a file per subscriber and it includes the coordinates at the end of the file. For each subscriber we should generate two CSV files with the timestamp and the Rx values in different columns.

antonroman commented 3 years ago

We also need to include the coordinates in the file, but maybe at the second line.

gbarreiro commented 3 years ago

There is a file per subscriber and it includes the coordinates at the end of the file. For each subscriber we should generate two CSV files with the timestamp and the Rx values in different columns.

What do you mean by generating two files? One for the hourly (S02) and other for the daily (S05) deltas?

We also need to include the coordinates in the file, but maybe at the second line.

Technically we can easily do that, but is it a good idea to include this information as a line not respecting the header defined in the first line?

antonroman commented 3 years ago

Hi Guillermo,

feel free to propose a better approach :-) maybe a file including the meter ID and its coordinates would be better. Thanks for the heads-up anyway.

in any case, If we already know that all of them are within a limited area this info is not so relevant and we can skip it to avoid data protection. We can analyze if it is interesting for the second part of the research, when we try to find anomalies and the location could be a meaningful variable.

gbarreiro commented 3 years ago

Then I'll skip the coordinates. The script is done, and I have the CSV files in my computer, but they weight a lot (14GB), so I'll show you the files tomorrow in our meeting and if they're okay, I'll compress them and upload them to Google Drive.

gbarreiro commented 3 years ago

As we had agreed in our last meeting, for the S05 values, now I will just pick the first row for each date (Ctr = 1, Pt = 0). I also realized there was a mistake in my code, so the S02 values were written into the S05 files too, but not it's fixed. Finally, I wanted to point out that during the JSON → CSV process, there are some JSON files which are not being converted because "something went wrong":

Screen Shot 2021-06-03 at 10 46 43

I've opened some of these files and in all of them, the timeline.s02, timeline.s04 and timeline.s05 lists are empty, therefore the error.

antonroman commented 3 years ago

Ok, how many files are wrong? sending a mail to Deicom is in my ToDo list :-) I can ask this as well.

Thanks!