abjer / isds2020

Introduction to Social Data Science 2020 - a summer school course abjer.github.io/isds2020
58 stars 92 forks source link

Problems with pd.read.csv #8

Open SofusBang opened 3 years ago

SofusBang commented 3 years ago

Hi everyone I am working on the weather data assignment, with importing weather data from 1864. I succeed in importing the data, and unzipping it using compression='gzip'

I think there is two problems: 1) The log command returns an error, thus suggesting that my data is not in the right format. I can use the log command on other data frames, so it is not the command itself

2) The observations I present using the iloc command does not correspond with the prober result, thus suggesting that there is something wrong with the ordering of the data

I post my syntax below

hederlist=['A', 'B','C','D','E','F','G','H',]
df_weather=pd.read_csv('ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/by_year/1864.csv.gz',names= hederlist,compression='gzip')
df_weather.head(n=10)
print(len(df_weather))
print(df_weather.iloc[-1:,0].values[0])

len=31921 df_weather.iloc[-1:,0].values[0]=USP00CA0003

the error with the iloc command ends with: TypeError: cannot do label indexing on with these indexers [0] of

Does anyone know how to solve this problem? Thank you in advance

jsr-p commented 3 years ago

Hi @SofusBang, when did you clone the repository? Have you seen this issue #1?

SofusBang commented 3 years ago

Hi @jsr-p

I tried to clone it yesterday this solved the problem of ‘iloc’ nor working. I still get different results when I run the assertion commands. Maybe it is the ordering of the data, because its seems like my data is ordered by date, and then observation alphabetically This is my syntax import pandas as pd import numpy as np

my_url='ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/by_year/1864.csv.gz'
df_weather=pd.read_csv(my_url, compression='gzip', header=None)
print(df_weather.head(n=70))
print(len(df_weather))

print(df_weather.loc[0,0]) print(df_weather.iloc[-1:,0].values[0])

And this is my output billede

Thank you in advance!

jsr-p commented 3 years ago

hi @SofusBang , I am not sure if I understand the problem. I get the same output when running your code and none of the assert statements raise an error. What error is it specifically that you get from the assert statements?

as_