search

deweyapp / dewey

Add custom tags, screenshots, and smarter search to your Chrome bookmarks.
deweyapp.io
Apache License 2.0
397 stars 51 forks source link

Screenshot not generated #145

Closed rlasjunies closed 8 years ago

rlasjunies commented 8 years ago

Hi all,

1st of all Great extension!

The screenshots are not generated on my installation. Looking in the code I feel the screenshots are generated calling this url 'http://dewey-server.azurewebsites.net/screenshot?query='

Unfortunatelly when calling it, I've got the answer 'Error 403 - web app is stopped'

Did I missed something?

outcoldman commented 8 years ago

@artemgrygor could you take a look?

artemgrygor commented 8 years ago

@rlasjunies @outcoldman it was hosted not on my primary subscription and for some reason it was disabled. Will try to setup on the main one with the same url.

artemgrygor commented 8 years ago

@outcoldman I can't setup a new instance with the same url because the previous subscription is not canceled. I've written a support ticket. I can create a new server on my Bizspark subscription but with the different url only. So we will have to update dewey app itself. Also @jamiewilson page2image doesn't work locally. Maybe it is a good idea to re-write application, at least, to work with "not show thumbnail" settings. Because right now it doesn't work too due to color-thief which is waiting for a screenshot to grab its main color. P.S. I will try to play with this one https://github.com/brenden/node-webshot also

artemgrygor commented 8 years ago

ok, thanks to MS support I recreated the subscription. So dewey-server is working now. But the screenshots no.

outcoldman commented 8 years ago

@artemgrygor thank you for doing that! screenshots are always a problem :( @jamiewilson your thoughts?

rlasjunies commented 8 years ago

Thank guys, The 'favicon' are shown in the thumbnail. Unfortunately not the screenshot (yet? ;-))

artemgrygor commented 8 years ago

@rlasjunies please switch off "Show screenshot" options in the settings menu while we're looking for the solution.

rlasjunies commented 8 years ago

Hi Guys,

I've spent few hours on the webshot. Phantom, used in webshot, seems to be very complex app. I did stupid browserify trial without success, I've reached a 'process.binding is not supported' error at early stage. I do not know Chrome Ext but I believe we will have issues for the "Node like" process used. For me, it does not smell good ;-( I would be happy to continue to dive in, but unless someone can show/guide me on this path, I do not think it's a good way to go.

Idea: can chrome ext. open new tabs, on the url we want thumbnail and take screenshot?

PS: I do not know Chrome Ext, Browserify ... so my analysis can be wrong

rlasjunies commented 8 years ago

Hi @artemgrygor, @jamiewilson, @outcoldman

Me again. I did several trialsss on the local screenshot generation.

The only way I found without a dedicated server is:

Pro:

Cons:

I will clean a little bit my trial. If you are interested let me know, I can create a new branch and share with you. Even if it's not as good than the server, one of the main feature of Dewey is still available.

Additionally I've some questions:

Richard

outcoldman commented 8 years ago

We thought about generating screenshots on client, but as you see it is not easy and requires additional clicks from user. Plus if computer will be slow (like chromebook) - it can work really bad.

Dewey is an "Chrome App". Is it something mandatory for you, can it be an "Chrome Extension"?

I believe there was a reason for that. Extension could not do something, which we need to do in App.

I would prefer an "Extension" that replace the "new tab" by default. Is it something you are against?

Take a look on https://github.com/deweyapp/deweylauncher

The query is made on the bookmark, not in the "Chrome app" list. Is it something you do not want?

Did not get this question.

outcoldman commented 8 years ago

@artemgrygor have you started to work on integrating node-webshot?

My thoughts about it If we are going to implement our own web server for generating screenshots:

  1. We should use https://developer.chrome.com/apps/app_identity to identify all requests to the webserver. This will help us to protect our server from misusing.
  2. On web server we should verify authentication token with google servers.
  3. We should cache all images and invalidate them not so often.
  4. We still should limit number of requests from client (we can actually do that with nginx proxy).
outcoldman commented 8 years ago

@artemgrygor ping

artemgrygor commented 8 years ago

@outcoldman @jamiewilson Omg, sorry guys. For some reason haven't answered this one. Unfortunately no, I'm quite busy right now. I have no experience with google identity will check what we can do. I agree with all these statements about caching. And also need to check where to store images. Should it be azure or amazon. On azure I have bizpark so we could use this one.

outcoldman commented 8 years ago

@artemgrygor np, I can take a look into it, when I will have a time, don't want us both to work on the same thing. Just wanted to check if you have started looking into it. I will try to build some prototype.

zeckdude commented 8 years ago

I get the following JSON Response when trying to hit http://dewey-server.azurewebsites.net/screenshot?query=https%3A%2F%2Fwebflow.com%2F as an example: {"status":"error","msg":"Call limit for today"}

That's happening with all my screenshots.

outcoldman commented 8 years ago

@artemgrygor do you have any stats - how many requests we were generating for the dewey users in hour / day? Also maybe how much traffic have we used on our proxy server?

artemgrygor commented 8 years ago

@outcoldman yes, I could probably get it from Azure stat but right now the portal is down :-) I can see web apps but not information about app itselft. I got lots of "Failed to load resource: net::ERR_CONNECTION_TIMED_OUT" Will check in few minutes.

artemgrygor commented 8 years ago

untitled

Ok, I got it. There were 458k requests with max 6k requests at one time in the last week. Important: this info with favicon too. So probably the screenshot numbers should be twice less.

outcoldman commented 8 years ago

458k in a week, right? 6k in an hour? About 2 requests per second. And I would assume it is about 50Gb in a week traffic, meaning 200Gb in a month. I guess simple droplet on DO can handle it. But that will cost $5 / month.

artemgrygor commented 8 years ago

Yes, looks like correct. Maybe free azure?

outcoldman commented 8 years ago

@artemgrygor up to you, as far as I know - Azure is not free :D But if you will host it with your account - that will be wonderful.

artemgrygor commented 8 years ago

@outcoldman right now we are using my free subscription, I don't pay for this. We can increase the plan and move it to my bizspark subscription if needed. Will have a look on DO too.

outcoldman commented 8 years ago

@artemgrygor so this is a plan I see:

  1. We will use https://developer.chrome.com/apps/app_identity to get the Google Auth Token for each user in the Chrome Application.
  2. On server we will use this https://developers.google.com/identity/sign-in/web/backend-auth#calling-the-tokeninfo-endpoint to verify that token is valid. Because this is external call - we will cache valid tokens for an hour or so.
  3. We will use https://github.com/brenden/node-webshot for generating screenshots.
  4. We can use http://redis.io for storing validated google auth tokens (we have exp field for it). Plus we can use it also for storing screenshots as well. Say we will expire them once in day/week.

To make a load on the server a little bit less - I guess we should remove url parameters.

artemgrygor commented 8 years ago

@outcoldman looks good for me. I don't know 1 and 2 so will have a look on your prototype. I'm ok with redis, want to work with it on nodejs. Also we can store favicon too. There are no restrictions from the google, as far as I know. But why not, it will increase performance too. And will expire them once in month probably.

outcoldman commented 8 years ago

@artemgrygor I have setup milestone v1 https://github.com/deweyapp/dewey-server/issues?q=is%3Aopen+is%3Aissue+milestone%3Av1 , let's create new branch dev, keep all work in separate branches and for all work create PR.

outcoldman commented 8 years ago

btw, if anybody wants to help with all of that - feel free to do that

outcoldman commented 8 years ago

I have deployed version 4.0.0 which is basically one big change - using our own screenshot generating server.

If you want to help: