Profile examples using urllib and numpy file reading

[matkrak]

Provide getting examples of profiles directly from the web using python urllib and numpy. Recieved data should be 2D or 3D array that can be directly used to create a profile with beprof tools.

[matkrak]

Should I add new python file like examples.py and write a script for getting data from URL and testing curve&profile (I could also make a separate class to store data and provide some tests as methods of this class) or I should rather figure out how to use urllib and put the tests in curve.py and profile.py ?

[grzanka]

I think that example using urllib should contain not more than 3-4 lines of the code. We can start working on it inside example.py file in separate branch. At the beginning of development it will be long and dirty. When its mature enough I'd like to put the code snippet in README.md file and remove example.py file.

What user installs ad pip package shouldn't contain any example. You can keep them in repo, but them you should filter the data to package in setup.py

First let's see how does it go this way. If there will be some surpises on the way I could change my mind.

[matkrak]

Already got familiar with urllib library, now I need to find a way to parse the data and what is going to be even harder: a nice example I could use. It probably wont take more then 2 hours. Also if you know some nice examples of profiles that could be used - let me know.

[matkrak]

Need some help here 1) Cannot find any csv/txt file with beam profile example (you provided some on the other branch but I have problem with parsing it) 2) I am using urllib.request.urlopen() as string generator, and I can read source of the page, but I can't parse it. Should I filter all non - numerical values first? 3) I am using numpy.genfromtxt() is this a good idea?

[grzanka]

Ad 1) Use your google skills. If you don't find the "beam profile" - try with another profile. It can be:

• daily temperature profile (its raising and falling, isn't it?). Using x_at_y and x_at_y(reverse) you can get time at which temperature is i.e. crossing zero degree point and calculate approximate time when we have positive degree.
• quantity of consumed alcohol during "Juvenalia" event (raising before, later falling) You can find many places which provide data in easily consumed format. See for example http://berkeleyearth.org/data/ (example dataset: "Monthly Average Temperature").

Ad 2) If you are parsing HTML - this is bad idea. I was thinking about sth like the example from here: http://stackoverflow.com/questions/6976685/using-numpy-to-create-yahoo-finance-price-table but much simpler (without need of converters etc):

import numpy as np
import pylab as pl
import urllib
url = "http://ichart.yahoo.com/table.csv?a=2&c=2011&b=30&e=7&d=7&g=d&f=2011&s=msft&ignore=.csv"
f = urllib.urlopen(url)
title = f.readline().strip().split(",")
data = np.loadtxt(f, dtype=np.float, delimiter=",", converters={0: pl.datestr2num}))

Ad 3) Yes. In fact you can even try to make it working directly with online files. Documentation says: http://docs.scipy.org/doc/numpy-1.10.1/user/basics.io.genfromtxt.html

If the argument is the URL of a remote file, this latter is automatically downloaded in the current directory.

[grzanka]

Check how it is done in pandas tutorial: http://tomaugspurger.github.io/modern-5-tidy.html

Loading data - 2 lines.