Load Data in GridApps?

[MaximeBaudette]

%cd ./data
    # load file into influx database
    output = !docker ps -qf "name=influxdb"
    container_id = output[0]
    path_to_load_file = str(path_to_load_file)
    !docker exec -t $container_id influx -import -path <$path_to_load_file> -precision s

In the container terminal, I tried to check the data as you told me to and executed:

select * from ieeezipload where time >= 1641024000000000000 limit 2

Which returns nothing.

[poorva1209]

select * from ieeezipload where time >= 1641024000000000000 limit 2 This command syntax is correct.

This is what I did to check the minimum and maximum timestamp in my container: for minimum timestamp: select * from ieeezipload order by asc limit 1; for maximum timestamp: select * from ieeezipload order by desc limit 1;

Then I ran the command as: select * from ieeezipload where time >= miminum_timestamp limit 2

[poorva1209]

To make sure you have ieeezipoad and weather datatypes in the influxdb, you can execute: show measurements which should show you a list as:

ieeezipload
weather

[MaximeBaudette]

HI Poorva, I can't find my matching timestamp. I believe something is wrong here: docker exec -t $container_id influx -import -path <$path_to_load_file> -precision s

Seems like maybe it didn't read the -precision s ? I'm absolutely not an expert on bash commands, and I scrambled to figure out the docker command

[poorva1209]

The load file (loadprofile_measurement_out.txt) need to be copied inside the container before we can import it. Try this:

docker cp loadprofile_measurement_out.txt influxdb:\ !docker exec -t $container_id influx -import -path loadprofile_measurement_out.txt -precision s

[MaximeBaudette]

I think that's what I attempted by using <path-to-file but I guess it didn't work like I wanted it to. Let me try that new approach

[poorva1209]

path-to-file should be the path of that file inside the container. Is that what you had?

[MaximeBaudette]

I have never really used docker, so I wasn't familiar with the fact you needed to first get the file into the container and only then put the local path from within the container.

try:
    %cd ./data
    # load file into influx database
    output = !docker ps -qf "name=influxdb"
    container_id = output[0]
    path_to_load_file = str(path_to_load_file)
    !docker cp $path_to_load_file $container_id:/$load_file_name
    !docker exec -t $container_id influx -import -path $load_file_name -precision s

this is now working I was able to verify the presence of the measurements directly from your influx query example !

I'll tackle the weather data now.

[poorva1209]

@MaximeBaudette I just pushed some changes to weather data upload in upload_update branch. https://github.com/GRIDAPPSD/gridappsd-data/tree/upload_update/timeseries/weather/solar-forecasting

Checkout upload_update branch and follow readme directions.

[MaximeBaudette]

It seems that the old script I used did manage to upload the weather data, however with the wrong time offset

[MaximeBaudette]

@poorva1209 I'd like to bring your attention to the fact that the weather data as provided is formatted as follows: Timestamp (%Y-%m-%d HH:MM:SS), Date(%m/%d/%y), time(HH:MM), [rest of the data]

Your bulk_load_weather.py script is aware of that double timestamping and:

• checks that there's 1 more column in the data compared to header
• offset all the data reading by 1 col to read the correct data.

I would recommend cleaning up these files, I can email you these files

[poorva1209]

@MaximeBaudette Will you be able to send the files today?

[MaximeBaudette]

Sent !

[poorva1209]

Closing this issue. I will create specific issues later based on discussions on how we can make custom loading of data better.