headzoo / surf

Stateful programmatic web browsing in Go.
MIT License
1.48k stars 159 forks source link

surf is incompatible with app engine #49

Closed ginabythebay closed 7 years ago

ginabythebay commented 7 years ago

I see two solvable problems in getting surf to work with app engine.

1. We need this commit brought into master: 40151144dcb8eb3fce7599cf59f7d2f50d1cf690. app engine is a sandbox environment and requires that any url fetching be done through their urlfetch package. They provide an http.RoundTripper implementation for this.

2. We need a way to disable the current agent code. On linux (which is what is used when building for app engine), the agent code expects to be able to use syscall, which isn't allowed in the app engine sandbox. Because surf.go calls the agent code when setting up its DefaultUserAgent var, there isn't a way around this without changing something in the surf library. For this, I propose adding an agent_appengine implementation that essentially returns some hard-coded values.

I'm glad to put together a PR for this. If you would be receptive to that, please let me know, and please let me know any thoughts you have about my proposed approach.

lxt2 commented 7 years ago

Fixed in master.