search

ishackm / Code-Bank

Hi, Welcome to my Code Bank. Here, you will find all my data science projects completed in university and in my hobbies.
2 stars 0 forks source link

Problems with template. #1

Closed vulcan25 closed 5 years ago

vulcan25 commented 5 years ago

Hello!

I think there is another template involved. base.html which this template overides. Is there any chance you could add that to this repo as a separate file so that I have the whole picture?

ishackm commented 5 years ago

Hi! The code you just saw now is the base.html, I will upload the index.html

vulcan25 commented 5 years ago

Perfect thanks. If you're using Flask I could demonstrate some code I built previousy which allows you to define the link bar items in Flask, and pass them to the template so that it builds the navigation bar dynamically. Please do upload the index.html file though, so I can see this too.

ishackm commented 5 years ago

Great. Thanks a lot. I have uploaded the index.html file also. I am using Flask-Bootstrap module on Flask. I have also uploaded the app.py file if you need it.

vulcan25 commented 5 years ago

Perfect. Give me a little while, and I'll upload something to demonstrate how to do this properly.

ishackm commented 5 years ago

Great. Thanks very much.

vulcan25 commented 5 years ago

Okay, so here's a really basic example which I've prepped in the time I had available.

This actually uses bootstrap v3.1.1 because my template code from previously was already based on this.

Here's a working example of the flat HTML: http://jsfiddle.net/gzmw1v9q/

You can see that it is responsive, in that it shrinks on mobile, and the dropdowns work.

I noticed you've tried to use the agency theme from startbootstrap. My example is actually based on a stripped down version of the grayscale theme from start bootstrap. Obviously an older version hence the 3.1.1 version of bootstrap.

Here's the repo: https://github.com/vulcan25/flask_nav_example

A few things to point out. When I looked at your templates there seemed to be <opening> and </closing> tags all over the place. Particularly <html>, <body>, etc. A correct HTML document in its most basic form looks like this:

<html>
<head>
    <title>My Website</title>
    <link rel='stylesheet' href='http://example.com/style.css' />
</head>
<body>
    <p>I am the homepage or something.</p>

    <script type='text/javascript' src='http://example.com/script.js'></script>
</body>
</html>

You need to make sure the tags are nested correctly.

Also CSS includes (in the <link> tag) belong in the <head> section. Obviously before the head section ends with the </head> tag.

Some HTML tags don't need a corresponding closing tag, the close statement can be put inside them, such as the break tag to take a new line: <br />. This is also valid syntax for the link tag which is why you see the /> at the end in the example above.

Javascript included by the <script> tags belong at the bottom of the page, ususally right before the body is closed with the </body> tag. You shouldn't use the quick-close shorthand for script tags though, they actually need the closing </script> even if they have a src attribute and don't themselves contain javascript. Of course this is also valid syntax:

<script type='text/javascript'>
// I am some javascript
alert('hello');
</script>

Ususally when you have a number of different script tags, you should have the one which includes jquery first, because the following scripts usually depend on it.

Jump into my app.py and you'll see how I've built the nav bar. I didn't actually use the Flask-Bootstrap extension, I just define my own python datastruture, and assign this as a flask config variable (app.config['NAVBAR']).

I then define a context_processor where I pass the navbar to the template:

@app.context_processor 
def inject_dict_for_all_templates():
    """ Build the navbar and pass this to Jinja for every route
    """
    # Build the Navigation Bar
    nav =[]
    for item in app.config['NAVBAR']:
        nav.append(item)
    return dict(navbar = nav)

This effects every route, so saves me having to pile the extra variables on individual render_template functions for each different route.

Then within the base template I have some nested for loops which make this data struture into the actual nav bar.

I'm going to stop explaining here, because I"m not really sure what your aim is to acheive.

I'd err on the side of caution with flask. depending on the functionality you need there may be easier ways to build this. If the majority of your site is going to be static, you may wish to go with something like jekyll which would give you this same nav-building functionality and supports the same template language as flask (I think - Jinja2). This would save running a flask server which is overkill for a static site. If you want some stuff with file uploads then you could still implement flask for this, or you could have the whole thing served by flask anyway- I dunno there's so many different ways to set things up.

vulcan25 commented 5 years ago

You'll see in my template aswell the index.html takes the most basic form:

{% extends "base.html" %}

{% block body %}

<p>Home page</p>

{% endblock %}

As I'm extending the base template, anything after the {% block body %} and before the next {% endblock %} replaces the corresponding block tags in the base template. (overides it).

So to create an upload page, you could just start with index.html (copy it to upload.html) but instead of <p>Home Page</p> have something else, which would then be surrounded by the rest of the base template.

Where it gets clever though: If you have a look at this section in the base.

This full {% block head %} {% endblock %} section sits within the actual html <head> tags. Don't get confused and think that overiding this replaces the <head> section. In actual fact it adds to it. Essentially this allows me to do the following in a child template:

{% extends "base.html" %}

{% block head %}
<link rel='stylesheet' href='http://example.com/style_sheet/for_index.css' />
{% endblock %}

{% block body %}

<p>Home page</p>

{% endblock %}

http://example.com/style_sheet/for_index.css will only be rendered within the specifc child template now, so that's useful for applying styles which should only affect one page/template. Of course because it's adding to the head section, any other stylesheets included in the base template will still apply to that child.

I've done the same thing with <title>{% block title %}{% endblock %}</title>

You could overide this block in any child template and the result would be a custom per-page title.

All you'd need to put is:

{% block title %} An actual title {% endblock %}

Obviously the correct nesting of tags is important though. In the child template, each block should end before another begins.

ishackm commented 5 years ago

Hi, Thanks very much for your feedback but how can I overcome that bug in the navigation bar. I am trying to create a database website so most of the pages would be not static. Is there anything that is wrong with the code below and can you show me how can I fix it please?

`<script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/4.1.3/js/bootstrap.bundle.min.js" integrity="sha256-E/V4cWE4qvAeO5MOhjtGtqDzPndRO1LBk8lJ/PR7CA4=" crossorigin="anonymous"></script>
<script src="/static/vendor/jquery/jquery.min.js"></script>
<!-- Plugin JavaScript -->
<script src="/static/vendor/jquery-easing/jquery.easing.min.js"></script>

<!-- Custom scripts for this template -->
<script src="/static/js/agency.min.js"></script>`
vulcan25 commented 5 years ago

At a glance, as I mentioned previously. Jquery should be included before the other JS files, as they will be reliant on jquery, so it needs to be loaded first. Something like:

<script src="/static/vendor/jquery/jquery.min.js"></script>

<script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/4.1.3/js/bootstrap.bundle.min.js" integrity="sha256-E/V4cWE4qvAeO5MOhjtGtqDzPndRO1LBk8lJ/PR7CA4=" crossorigin="anonymous"></script>

<script src="/static/vendor/jquery-easing/jquery.easing.min.js"></script>

<script src="/static/js/agency.min.js"></script>`

I'd also open up your browser's dev tools once you've made these changes and confirm that none of the static includes you're hosting locally from your /static directory are giving a 404.

Another thing. Probably worth making sure that any bootstrap CSS includes (again which should be in the head section, not at the bottom of the page like the JS includes) match the version of your bootstrap JS.

So in the code section right here you've picked 4.1.3. I'd Just pull the matching CSS version from the CDN, and replace any link to bootstrap.css which you have in your head section:

<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/4.1.3/css/bootstrap.min.css" integrity="sha256-eSi1q2PG6J7g7ib17yAaWMcrr5GrtohYChqibrV7PBE=" crossorigin="anonymous" />

IF you need to find different versions for yourself ...

As you might have noticed you can load https://cdnjs.com/libraries/twitter-bootstrap/4.1.3 in your browser, which gives you a list of includes for this version. Don't include that link itself in your template.

To find what you need to paste in, hover at the right of the page and choose COPY -> Copy Link Tag with SRI. This gives you the block I've just pasted above. Obviously make sure it has the correct filename, bootstrap.min.css in the example above.

ishackm commented 5 years ago

Great Thank you, after many hours of trying, it finally works! If I have any further questions, is that ok with you?

vulcan25 commented 5 years ago

I'm happy I could help. Please feel free to ask further questions 👍

ishackm commented 5 years ago

Hi there!

I hope you're well.

I just have a few more questions about Flask, please.

In my website, I have to create a function where the user uploads an excel spreadsheet and then imports the data into matplotlib to create some graphs, which then will be shown on the website.

I have already created an upload function in Flask and also created the matplotlib graphs in Jupyter notebooks. I just need help combining them together in my app.py and also show the images on the website.

Do you mind helping please and do you know any good resources to achieve this aim?

Many Thanks,

ishack

vulcan25 commented 5 years ago

One question I have... Which operating system are you using on your development box by the way?

Also can the user's upload the data in CSV instead of EXcel, is that possible? And what kind of charts are you making, line, bar or something else?

ishackm commented 5 years ago

I am using Mac OSX

Yes, the user can upload in the csv file format and the graphs I have made is mostly bar charts and some pie charts. I have also upload app v2.py for you to see.

vulcan25 commented 5 years ago

Thanks, you might be interseted to take a look at my personal project, which is to log temperature data submitted from a raspberry pi. I used the ChartJS library to begin with which can be seen here:

https://ser.v25.uk/d0ac059c370ef464

This has support for a wide variety of charts, including Pi, bar etc: https://www.chartjs.org/samples/latest/

However I found that with a large number of datapoints, this started to cause problems (slow rendering time of the chart) so I also looked at Dygraph, which only supports line graphs, but handles large ammounts of data in a better fashion. The dygraph version can be seen here:

https://ser.v25.uk/d0ac059c370ef464/dy

The documentaion for this one is here: http://dygraphs.com/

However as you can see, I had some problems with the implementation of this (one line falls from the bottom of the chart, which I haven't solved yet). Dygraph actually loads the information in a CSV format (which my data was not - so I'm still strugling with converting the data from the chartjs compatible format so to speak).

I haven't actually worked with matplotlib or jupyter. To sumarise the above:

Chart.js : varied chart types, and good for small ammounts of data ( < 5000 points per chart) Dygraph: handles large sets of data (millions of points) well, but limited to line charts.

As for your implementation there's a few parts to the process, and questions you'd need to ask yourself...

If you're using mac os, then that's great. If you haven't already you could download the git client https://git-scm.com/download/mac and make an acutally repository with all your code which you could push to github (unless you have an issue with that?). This would help me out if I could see all the code you'd written, rather than just one off text file uploads. IF you need some guidance on how to do that, please let me know.

Any thoughts?

vulcan25 commented 5 years ago

Of course if you had the matplot lib code developed, and modular enough that it could take a CSV file, and output an image file, all you'd need to do is output the image to the filesystem, and then have flask display said images on the site. This wouldn't have the interactive (pretty) features of the other chart libs though. Again if I could see the whole code, then I could give better advice.

ishackm commented 5 years ago

Hi, Thanks for the info.

Unfortunately, we are only supposed to use matplotlib for this project and the data points are 10,000 plus. Also, I am making a lot of bar charts so I think matplotlib would be more suitable. Right now I'm testing the matplotlib code on Jupyter and I hope to upload it tomorrow.

Many Thanks,

Ishack

vulcan25 commented 5 years ago

Sounds good, look forward to seeing it.

ishackm commented 5 years ago

Hi, I have uploaded the templates and the matplotlib script. I am trying to run app v2.py but its giving me an error. I would like to combine it with the matplotlib code so that the when the user uploads a csv it creates graphs and then shows them on the website, please?

What's the best way to achieve this aim?

vulcan25 commented 5 years ago

Can you send up the sample CSV file also so I can run this code?

ishackm commented 5 years ago

Sure it's done sample.csv

vulcan25 commented 5 years ago

Also, Did you manage to run this code yourself (the matplot lib part) if so what was the output of this, can you provide a screen shot or similar of what the results of this code were on your machine? Preferably with the same sample.csv inputted..

Also there are a number of other csv files in the code, can you explain what their functionality is. Do these come from the user or already exist on the system?

9:filename = input("Upload your own data: "); #az20.csv (must be a csv file, flask does not support tsv)
39:uniprot_kinases=pandas.read_csv("https://www.uniprot.org/docs/pkinfam.txt", sep="\t", names=["Column1"],skiprows=74, skip_blank_lines=True )
62:protein_alt_names=pandas.read_csv("Protein_alt_names.csv", header=None)
67:kinase=open("kinase_info_final.csv","r")
102:newfile=open("Protein_alt_names.csv","w+")
119:info4.to_csv("az_filtered.csv", sep=",")
ishackm commented 5 years ago

39:uniprot_kinases=pandas.read_csv("https://www.uniprot.org/docs/pkinfam.txt", sep="\t", names=["Column1"],skiprows=74, skip_blank_lines=True )

This is a csv parsed from online

62:protein_alt_names=pandas.read_csv("Protein_alt_names.csv", header=None)

I have uploaded this csv

67:kinase=open("kinase_info_final.csv","r")

I have also uploaded this file

All these csv files conduct data analysis of the orginal az20.csv and produce graphs from matplotlib.

The graphs produced:

img-20190129-wa0007 img-20190129-wa0008

vulcan25 commented 5 years ago

is az20.csv == sample.csv on your upload?

I only get the one graph outputed

1

Is this what should be expected when inputing this sample.csv at the prompt?

ishackm commented 5 years ago

Yes that's exactly what the output should look like.

is az20.csv == sample.csv on your upload? Yes that's right

vulcan25 commented 5 years ago

Thanks,

What I'm trying to understand in this code....

The graph I posted to you appears to be generated here, around line 128 which is the last in this block:

substrate = info4['substrate']
fold_change = info4['az20_fold_change']
plt.scatter(substrate, fold_change, edgecolors='b')
plt.xlabel('Substrate')
plt.ylabel('Fold Change')
plt.title('Scatter Plot')
plt.show()

What about the remaining code that comes after it.? why is this not executed, and what should it do?

ishackm commented 5 years ago

For now, please remove it because for now i just want it to show the image on Flask after the user uploads the file

vulcan25 commented 5 years ago

Thanks if it's okay I will take until tomorrow to investigate this fully. Please stay tuned I think it should work with some small modifications.

ishackm commented 5 years ago

Yeah great Thanks very much!

vulcan25 commented 5 years ago

DISCLAIMER: This aint production ready yet. Consider it a proof of concept. Probably read this whole thing before attempting. This has been good to confirm my own learning aswell, it's become a bit of a tutorial. I've submitted the matplotlib module you provided to my repo (I hope you don't mind, can ice it if you do). You'll need to import all the txt/csv files yourself to the working directory as I haven't included these. Again read this whole thing before commencing.

A quick word of advice before we start. This should be possible on a mac if you've got python3.

You can run these commands in a fresh directory:

python3 -m venv graph
source graph/bin/activate

Now you have a python virtual environment which keeps anything installed with pip isolated to the graph project. This would be indicated by (graph) at the start of your shell. I've packaged a req.txt file that contains everything you need to run this. You can install those requirements with:

pip install -r req.txt

Then at any point, install more things with pip install whatever-package if required.

Then to export what's installed in the venv back to req.txt run:

pip freeze > req.txt

It's the way to go!

As you'll see shortly, the flask_nav_example repo from earlier is completely updated, and works minimally. I strongly suggest you create a new venv from the included req.txt and treat this separately from what you've been working on.

I also had to rename matplotlib.py to mat.py because I installed the module matplotlib which would clash if I have a python file of the same name (.py)

How I solved this...

I could see that plt.show generates the graph, and as you mentioned, commented out everything below this as it's irreverent.

Quick google search for plt show output png reveals:

https://stackoverflow.com/a/9888817/2052575

This must be commented out:

plt.show()

And replaced with:

plt.savefig('foo.png')

savefig() is a matplotib function which can also be passed foo.pdf to change the output file format, but we'll stick with png format for now.

I run the code again and at the prompt feed it the sample.csv Then confirm that foo.png has been created on the filesystem and contains the expected chart.

This line becomes the end of the code block (everything below is commented)

Next I look at the top of the code:

filename = input("Upload your own data: "); #az20.csv (must be a csv file, flask does not support tsv)

info=pandas.read_csv(filename, sep="\t")

All we are feeding to this is the filename. So we can make the whole block of code more modular by defining a function, and indenting everything below, down to plt.savefig() so it becomes part of that function. I also pass in an output variable which i then pass to plt.savefig() so that when we call proccess_image I can define the name of the output file name. It now looks like this:

filename = input('Upload your own data:')

def process_image(filename, output):
    #
    # [SNIP] Rest of processing code
    #
    plt.savefig(output)

Now I can call the function with process_image(filename, 'output.png')

I also comment out the line filename = input ('upload your data') as we don't want that interactivity happening when we call this as a module.

I also downloaded pkinfam.txt to the working directory, and changed that line in mat.py to:

uniprot_kinases=pandas.read_csv("pkinfam.txt", sep="\t", names=["Column1"],skiprows=74, skip_blank_lines=True )

This avoids the URL, and also means I can comment out import urlib3 at the top which will make this run better under flask (we don't want to be grabbing remote URLs on every request.)

As for the flask side, I had some problems running your app as the templates crashed again. So I've just used the code I previously had shown you (flask_nav_example). I added flask-restful support and an upload endpoint that I had previously tested for someone else.

I move the csv files (and the pkinfam.txt which I had previously downloaded to save including urlib3) into my working directory, then I can simply test with curl:

curl -i -X POST  -F 'file=@sample.csv' "http://localhost:5000/upload" -H 'ContentType: multipart/form-data'

Here's what the upload code looks like at this point: 370ecc548e35ee7b3d7dda58824908bfb8e502eb

As you can see this just puts the file into the following folder:

UPLOAD_FOLDER = 'static/img'

I can confirm that with:

$ ls static/img 
received.png

Oh shit, that's actually a CSV which has been called received.png cause that's what's hard-coded into the script. Never mind though we can proceed to sort that out:

Let's take a quick look at the panda.read_csv documentation to work out what that function accepts. Remember that's what our process_image function we created earlier is sending the filename to.

filepath_or_buffer : str, path object, or file-like object Any valid string path is acceptable. The string could be a URL. Valid URL schemes include http, ftp, s3, and file. For file URLs, a host is expected. A local file could be: file://localhost/path/to/table.csv. If you want to pass in a path object, pandas accepts either pathlib.Path or py._path.local.LocalPath. By file-like object, we refer to objects with a read() method, such as a file handler (e.g. via builtin open function) or StringIO.

Sounds hopeful. Let's see if we can make Flask pass the uploaded file straight to our function.

I import the function we made earlier like so:

from mat import process_image

Then a quick edit to the FileUpload object:

#photo.save(os.path.join(UPLOAD_FOLDER, filename))
process_image(photo,'output_from_flask.png')

remember photo here is actually the csv file!

Try the above curl command again, and if you're playing along you'll see lots of matplotlib info scroll past on the flask development server and boom: output_from_flask.png exists and it contains the chart!

I'll make some changes (mainly rename that pesky photo var to csv for clarity) and that's us at commit c3c8b49c551c7a50ac455d38461ae02b471fcf29 On the next commit I added the req.txt.

PROBLEM: Just got this far and discovered that even though the photo uploads and saves to the local directory, the Flask application immediately crashes. That's not good, here's the error:

Python[26803:1005902] WARNING: NSWindow drag regions should only be invalidated on the Main Thread! This will throw an exception in the future. Called from (

Quick search reveals:

https://github.com/processing/processing/issues/5676 https://stackoverflow.com/questions/52178375/flask-function-return-invalidated-on-the-main-thread-error

At a glance could be a Mac Mojave bug. Don't know if we'd get around this buy using a docker container. (Hint from the future: this will work).

Get docker for mac: https://docs.docker.com/docker-for-mac/install/

If you're gonna do it this way you don't technically need to make the venv locally as I mentioned at the start, as all this will live inside docker.

Here's the Dockerfile:

FROM python:3.7.2-stretch

WORKDIR /usr/src/app

COPY req.txt ./
RUN  pip install --no-cache-dir -r req.txt

CMD ["python","-m","flask","run","--host=0.0.0.0"]

Let's build the docker image and tag it mat-app-image, then confirm it's there:

docker build -t mat-app-image .

No errors should come up here. It should grab all the requirements, etc. Then check it exists"

$ docker images
mat-app-image       latest              c1a5fc3be2c4        9 seconds ago       1.18GB

Sweet. Only slight issue here is it's a pretty large image. the python:alpine image is much smaller, but because of the dependences (matplotlib, etc) I thought stretch was a less troublesome match for the size. Now run the docker image:

docker run --name mat-app-runtime -p 9500:5000 -v `pwd`:/usr/src/app mat-app-image:latest

--name mat-app-runtime : Here we're giving the running instance a name -p 9500:5000 : maps port 9500 on your machine to 5000 within the container

-v `pwd`:/usr/src/app

This line maps a volume from the current working directory to our app directory within the container. pwd in the backticks just grabs the current working directory.

finally mat-app-image:latest refers to the image we built earlier.

You should see the flask development server run at this stage. IF you want it to go in the background, just CTRL+C then you must docker rm mat-app-runtime before executing the docker run command again, this time with the -d option to daemonise it.

Then confirm it's running with docker ps:

bd8b4b088263        mat-app-image:latest   "python -m flask run…"   4 seconds ago       Up 2 seconds        0.0.0.0:9500->5000/tcp   mat-app-runtime

Now from another terminal, hit it with curl, first removing the previous output_from_flask.png which would have been created:

rm output_from_flask.png

curl -i -X POST  -F 'file=@sample.csv' "http://localhost:9500/upload" -H 'ContentType: multipart/form-data'

result:

HTTP/1.1 100 Continue

HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 16
Server: Werkzeug/0.14.1 Python/3.7.2
Date: Thu, 31 Jan 2019 13:37:39 GMT

"file uploaded"

And output_from_flask.png should now have been created :D

At this stage I commit the actual mat.py and the Dockerfile.

You can clone the repo in it's current form from https://github.com/vulcan25/flask_nav_example and do the docker steps above to make it work. Remember to copy in the text/csv files to the root direcotry.

I'm gonna stop here just now. Next steps:

vulcan25 commented 5 years ago

I've just been pondering other ways to impement this. Obviously in it's current implementation there's some security risks because there's no validation or authentaction on the upload endpoint (as I said proof of concept! . I've been designing some code to do this, which I haven't pushed yet, but it's always a grey area, they say 'don't re-invent the wheel'.

Think about your user base. Who are they? Do they all have github accounts for example?

Could the user's push their source .csv files to a github repo for example, then there might be a way to use the github API from your flask app, so when someone accesses your site it will:

This means you're not worrying about designing the upload functionality yourself, if you follow me.

Of course if this is just for some internal kind of app them maybe the custom upload system would work, I dunno...

ishackm commented 5 years ago

Hi, thanks for your detailed research! Very advanced for a beginner like me! I just read a relatable tutorial so I was wondering if it would be possible to do the same with matplotlib?

https://github.com/viveksb007/camscanner_watermark_remover

This project is rather about the functionality than security at the moment.

vulcan25 commented 5 years ago

Yeah, this code is a pretty similar idea.

Where they run they remove_watermark() function in app.py is probably where you want the functionality which is in the mat.py module of your code.

Rougly, you could:

from mat import process_image

At the top of the script. And then within this function:

def process_file(path, filename):
    remove_watermark(path, filename)

Call process_image() instead of remove_watermark.

This is essentally the stuff I've outlined above. When I moved on to docker this was due to the bug thrown by running the matplotlib stuff within the flask app on OSX, so I'm not sure if you'll stumble on this yourself.

vulcan25 commented 5 years ago

Hopefully the stuff I've put above isn't too confusing. I guess the best way to approach it is at your own rate, and you can always refer back later if needed. I did just want to point out that making something which is internet accessible has a whole other host (no pun intended) of considerations. Running the app with python app.py for the dev server tends to be replaced by something like gunicorn+nginx in production, due to performance reasons among other things, which could introduce other bugs not spotted in the development stage. I haven't even covered that part yet lol.

Good luck to you, and if you have any further queries, please don't hesitate to write!

ishackm commented 5 years ago

OK thanks, I will give a try and get back to you soon if I get any errors, if you don't mind

ishackm commented 5 years ago

Hi, I edited the Camscanner example and uploaded the folder as matplotlib upload , can you please run it and advise on how I could export the graphs to a particular directory and show it on thee Flask website. I also improved the code of the matplotlib so now you should see two graphs.

ishackm commented 5 years ago

Also, its az20.txt rather than csv, the code works better with txt.

vulcan25 commented 5 years ago

I've had a look at the new file matplotlib_upload/app.py and I suspect you haven't fully understood what I wrote before, particularly the part about creating the function process_images() and laying that code in mat.py out like this:

def process_image(filename, output):
    #
    # [SNIP] Rest of processing code
    #
    plt.savefig(output)

Maybe I didn't explain the reasoning behind this. This is quite important so let's disect this application:

from flask import Flask, render_template
app = Flask(__name__)

@app.route('/')
def index():
    return render_template('index.html')

if __name__ == '__main__':
    app.run(debug=True)

What's actually happening here?

First you import the Flask class, and the render_template function. The second line here, creates an instance of the Flask class (we assign that to app in the following code. It is true to say that app is now our application object):

from flask import Flask, render_template
app = Flask(__name__)

Then we define a function called index. We use a decorator (signified by the @ which is a python thing) provided by the Flask library, to add a route to this function. The route decorator belongs immedaitely above the def statement which defines the function. The app. part at the beginning of the decorator means the decorator applies in the context of the app object we created earlier.

@app.route('/')
def index():
    return render_template('index.html')

The '/' passed in the brakets - the first argument to the route decorator - specifies the URL or 'endpoint' which will be mapped to the index function. In laymans terms when a user hits the root URL (/), the index function will run and return the rendered template (index.html in this case).

At this stage, you have a valid flask application. You could put these 5 lines in any file (for example run.py or ishackm.py, and it would be true to say you have a run or ishackm module, containing an application object called app.

You then have the ability to import the application object from somewhere else, using a statement like:

from ishackm import app

It doesn't matter that, in the original file, the @app.route part came after you set app = Flask(__name__) because this isn't procedural. Wherever you do this import the app's context will have the route decorator implied.

This may not seem important at the moment, but in the future it will be, because there are numerous advantages to being able to pass the application object between modules. One practical use of this is when it comes to serving in production. You'll probably end up using a tool like gunicorn which can serve WSGI applications. It just so happens that app (our application object) IS a UWSGI application already!! Gunicorn would therefor be pointed directly at ishackm:app.

Sometimes confusion is added because people put all of this in a file and call it app.py which is differnt from the app contained inside the object. People (I'm bad for this aswell) then use the two terms interchangably. This is made worse by the common usage these days of 'app' as an abbreviation for 'application' where years ago was actually a 'computer program' :D. To to summarise:

So where am I going with this? Well, what would happen if you wrote the first 5 lines and ran python ishackm.py on that version of the file? Nothing! Because all you've done is defined an application object, and a decorated function. So that's where the last 2 lines come in:

if __name__ == '__main__':
    app.run(debug=True)

When you execute this file from the command line, with python ishackm.py the interpreter automatically sets the variable __name__ to "__main__" so the if statement returns true, and the following line executes. *Alternatively when you import ishackm from somewhere else,__name__ will be set to "ishackm" so the following line won't execute.

So that makes this a good place to execute the development server in, which is what the run method of our application object does. debug=True part is obvious.

Essentially app.run() is a method which gives you this output at the terminal:

 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
 * Restarting with stat

You don't want that happening in production because the development server's crap, but ALSO because the production server (gunicorn) is designed to load the application object directly, rather than executing the run method which writes the above data to stdout.

So back to matplotlib_upload/app.py and what's wrong here.

The main problem is you've just pasted the matplot logic straight into this file, at the same level as all the Flask code. What this means is if you lauch the application (either via python app.py or using import app from app) it will execute this matplot code immediately.

To illustrate this, if I add this line print ("OH DEAR") on line 1 of my app.py, and run the development server, I'll see:

$ python app.py
OH DEAR
 * Serving Flask app "app" (lazy loading)
 * Environment: production
...

Similary if I try to import my application object, in this case with the python terminal:

>>> from app import app
OH DEAR
>>>
# Point to note, I could actually execute app.run() right here.

This is a problem for you, because you don't want the matplot logic to be executed when the application object is imported, or the development server runs.

You want it to be in the proccess_image() function, which can be passed specific arguments, so that it can be called modularly when required by your computer program ;-)

That way, when it eventually gets tied to an endpoint, via a route decorator, it can be run on demand when someone uploads an image, rather than once when the program starts.

--

I've pushed some form code to my repo, which has upload functionality implemented in javascript with a spinner: https://github.com/vulcan25/flask_nav_example

I've still to implement the directory organization and frontend display of charts, but I'd say this is probably a good imeplmentation so far, based on the points mentioned above.

ishackm commented 5 years ago

Oh I see, so the matplotlib code goes after the process.image() function?

vulcan25 commented 5 years ago

Can you explain this...

    kinase=open("kinase_info_final.csv","r")
    line=kinase.readline()
    alt_names={}

    while line != "":
        line=kinase.readline()
        line=line.split(",")
        if len(line)<2:
            break
        elif line[2] not in alt_names:
            alt_names[line[2]]=[]
            genenames=line[3].split()
            for y in genenames:
                if y not in alt_names[line[2]]:
                    alt_names[line[2]].append(y)
            line2=line[-1].split("(")
            del line2[0]
            for y in line2:
                y2=y.replace(")","")
                y3=y2.replace("\n","")
                while True:
                    if y3[-1]==" ":
                        y3=y3[:-1]
                    else:
                        break
                if y3 not in alt_names[line[2]]:
                    alt_names[line[2]].append(y3)

    for x in alt_names:
        if x=='':
            del alt_names[x]
    kinase.close()

    newfile=open("Protein_alt_names.csv","w+")
    for x in alt_names:
        for y in alt_names[x]:
            newfile.write(x+","+y+"\n")
    newfile.close()

What's the origin of kinase_info_final.csv

Am I correct in understanding that the logic in the above code processes this and writes it to: Protein_alt_names.csv

So would it be correct to say thatkinase_info_final.csv would never have to be read again once this logic is completed, and to make the code more efficient the end application would only ever require Protein_alt_names.csv

Or is kinase_info_final.csv subject to change?

vulcan25 commented 5 years ago

Also:

    protein_alt_names=pandas.read_csv("Protein_alt_names.csv", header=None)
    protein_alt_names

This block appears above the block in my previous comment, but protein_alt_names doesn't appear to be used again in the code. Unless I'm misunderstanding the significance of the second line here.

It looks like this is just reading the file into panda's and printing the info out?

ishackm commented 5 years ago

Hi, the kinase file has all the data on all human kinases and the protein file has all the protein names. So yes the kinase file is only used once.

If you run the matplotlib code on Jupyter or similiar , it wil make more sense.

ishackm commented 5 years ago

Yes thats right, it just prints out all the info

vulcan25 commented 5 years ago

Oh I see, so the matplotlib code goes after the process.image() function?

The matplotlib code (your custom logic) goes IN the process.image() function.

That's what I meant by this part:

def process_image(filename, output):
    #
    # [SNIP] Rest of processing code
    #
    plt.savefig(output)

"Rest of proccessing code"is what was originally in your matplotlib.py file. This is a more modular way of writing it so that you can call it with:

process_image( INPUT, 'graph.png')

Where:

ishackm commented 5 years ago

Hi, I have uploaded a new version of the uploadfinal.py. Can you have a look, please? I am slowly understanding the concept of what you're trying to say but I can't seem to put it together in code.

Can you please comment the code if it's not too much and show me how to integrate with app.py?

vulcan25 commented 5 years ago

Okay, looking at uploadfinal.py:

My editor is sublime 3 which is good and free, so I can do this by highlighting those lines, and pressing TAB (indent) and SHIFT+TAB (unindent). To move the whole block at once.

Here's a video of that: https://imgur.com/a/k4s6Yvb

Why is this sinificant...

Indentation in python is very important. Unlike javascript where curly brackets define stuff that's blocked together.

So, in Python:

def hello():
    print (1)
print (2)

is not the same as

def hello():
    print (1)
    print (2)

Also, assuming this code was in a module (a file called sample.py) then, in another module you put import sample:

To illustrate another way I quickly create a file using the terminal:

air >  flask_nav_example git:(master) ✗ cat > sample.py
# A quick way to make a new file
print ('I am not indented')
def business():
    print ('Taking care of business')
^C

Then demonstrate with the python shell:

air >  flask_nav_example git:(master) ✗ python
>>> from sample import business
I am not indented
>>> business()
Taking care of business
>>> sample.business()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'sample' is not defined
>>>

Can you see why the last execution throws an error? sample.business() can't be called directly here. For that I should have done import sample. Either way though I am not indented will always print the second I do the import.

So back to your code. Once you've indented lines 40 -> 225 it begins to look like this:

fixed slightly

So assuming we keep the logic in the main uploadfinal.py file, and nuke that Line 4 and 38, here's what it now looks like:

mod

Looking better, but a couple of problems still exist here:

So where are we supposed to call this form? The index() function just above. Currently that line looks just like this:

process_file(os.path.join(app.config['UPLOAD_FOLDER'], filename), filename)

This ought to look more like this:

process_file( 'output_test.png', file)

I also comment Line 31 as it's not required, and we've got something liek:

modified more

Round about line 167, you need to remove plt.show() cause that's again an interactive feature which launches a rendering of the chart in your operating sytem's gui. Obviously not ideal in a flask application, we want to save that image to disk. If you search my previous post, for plt.savefig you'll see the way this is done, so given that you're passing a string 'output_test.png' to process_file() as the path argument this needs to look like:

plt.savefig(path)

Remember to respect the indentation. That needs to remain within the process_file() function, so should remain indented.

I'm going to stop here. Hopefully the above helps you, I realise it's not fully complete but I'm at a fork where I'm in danger of rewriting some stuff that's already done correctly in my flask_nav_example code.

Perhaps could you give me some feedback on this post. I'm hoping that it's made a few things clearer, and you may then be able to look at flask_nav_example with more knowledge of what's happening and why.

Particularly there may still be bugs based around the index function in this code and the file upload stuff, it can get slightly complicated with all the references to UPLOAD_FOLDER etc. Also I don't have your exact templates for this version.

A few final thoughts of this post:

Currently all this is within the process_file() function:

import pandas
import urllib3
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from pandas.plotting import _converter

You don't want that there: it will run these library imports every time process_file is called. These belong at the very top of the file so the stuff's imported once, when the flask application runs. As it stands this would cause performance issues.

ishackm commented 5 years ago

Hey Thanks for the commenting through the code, its much clearer now. Do you mind uploading the code that you showed the previous post? I will experiment with it and get back with you asap.

Thanks,

Ishack

vulcan25 commented 5 years ago

show me how to integrate with app.py

Everything in this post assumes you're executing python uploadfinal.py because in there you have defined the flask application, and that's where all the routes are so uploadfinal.app is a valid Flask application object.

Again with the stuff I've gone over, you'll see why in flask_nav_example i put the matplot lib code it's own file called mat.py and then in my app.py I:

from mat import process_image

then just call the function with process_image() further down the file.

This has the effect of keeping the maths logic separate from the flask logic.

vulcan25 commented 5 years ago

Hey Thanks for the commenting through the code, its much clearer now. Do you mind uploading the code that you showed the previous post? I will experiment with it and get back with you asap.

Hopefully no sync errors between this and what I posted.
I've tried to keep the line spacing as similar as possible.

https://gist.githubusercontent.com/vulcan25/44419bcb8daa9748be4c9abfcef5e1ed/raw/2bd914db89cf71255763ead22c1f6f99f5569c6b/gistfile1.txt

vulcan25 commented 5 years ago

Just testing something here, will add more description shortly...