search

uw-echospace / batmo

0 stars 0 forks source link

Write a script that pulls this years data #4

Closed m-ball3 closed 1 year ago

m-ball3 commented 1 year ago
m-ball3 commented 1 year ago

Base script is written. Pulls hourly (easily changed to daily). Need to resolve some issues with it:

  1. formatting in excel file is bad (headings are not aligned with data points, weird data points in first row)
  2. when attempting to fix formatting issue, gets error message in excel (too much data)
  3. need to figure out how to get data every 15, 30, or 60 minutes instead of every minute I want to make sure that this yearly data is in the same format as the 2022 data is in
leewujung commented 1 year ago

Base script is written. Pulls hourly (easily changed to daily). Need to resolve some issues with it:

  1. formatting in excel file is bad (headings are not aligned with data points, weird data points in first row)
  2. when attempting to fix formatting issue, gets error message in excel (too much data)

Maybe try using csv files?

  1. need to figure out how to get data every 15, 30, or 60 minutes instead of every minute I want to make sure that this yearly data is in the same format as the 2022 data is in

If the data format changes over time, you could settle down on a particular one and figure out the difference between the years and "convert" time. This is the "devil's in the detail" type of data wrangling -- welcome to the club! :)

For pulling data regularly, you could try Windows task scheduler and write a .bat file to run a .py file (that's the way I made it work on the ship, with lots of googling). You can also set up a cron job through "crontab" to run on the linux computer in the lab. Now that you are familiar with command stuff, that may be a fun way to use the skills!

m-ball3 commented 1 year ago

I am using .csv files, but I think I resolved the issue! I made a couple of modifications this morning and the .csv I got from it looks good so far!