Throw a more informative exception on missing credentials

[bzg]

Hi there! Thanks again for twttr. I'm using it for this small Twitter bot.

I tried to dockerize the application and stumbled upon an issue: how to make sure aleph is listening to the 0.0.0.0 host? The issue is explained in the context of a ring application here:

Caution: your Ring web server should be configured to listen on 0.0.0.0 address in order to play nicely with docker.

I assume this is the issue I have, I'm not 100% sure. Below is the log I get when running the dockerized version of the application.

09:53:48.664 [main] DEBUG io.netty.util.internal.logging.InternalLoggerFactory - Using SLF4J as the default logging framework                                                
09:53:49.040 [main] INFO eigforever-twitter-bot.core - Listening to tweets...                                                                                                
09:53:49.042 [main] INFO eigforever-twitter-bot.core - Following EIG users                                                                                                   
Exception in thread "main" java.lang.NullPointerException                                                                                                                    
        at java.net.URLEncoder.encode(URLEncoder.java:204)                                                                                                                   
        at oauth.signature$url_encode.invokeStatic(signature.clj:101)
        at oauth.signature$url_encode.invoke(signature.clj:97)
        at oauth.signature$fn__5628.invokeStatic(signature.clj:65)
        at oauth.signature$fn__5628.doInvoke(signature.clj:63)
        at clojure.lang.RestFn.invoke(RestFn.java:442)
        at clojure.lang.MultiFn.invoke(MultiFn.java:238)
        at oauth.client$credentials.invokeStatic(client.clj:90)
        at oauth.client$credentials.doInvoke(client.clj:79)
        at clojure.lang.RestFn.invoke(RestFn.java:533)
        at twttr.auth.UserCredentials.auth_header(auth.clj:76)
        at twttr.middleware$assoc_auth_header.invokeStatic(middleware.clj:63)
        at twttr.middleware$assoc_auth_header.invoke(middleware.clj:55)
        at twttr.middleware$wrap_auth$auth_middleware_handler__6943.invoke(middleware.clj:75)
        at twttr.middleware$wrap_body$body_middleware_handler__6936.invoke(middleware.clj:51)
        at aleph.http$fn__5428$request__5432.invoke(http.clj:211)
        at twttr.api$request_endpoint.invokeStatic(api.clj:103)
        at twttr.api$request_endpoint.invoke(api.clj:89)
        at twttr.api$fn__7010$fn__7012.doInvoke(api.clj:125)
        at clojure.lang.RestFn.invoke(RestFn.java:439)
        at eigforever_twitter_bot.core$statuses_filter_eig.invokeStatic(core.clj:31)
        at eigforever_twitter_bot.core$statuses_filter_eig.invoke(core.clj:26)
        at eigforever_twitter_bot.core$retweet_eig.invokeStatic(core.clj:46)
        at eigforever_twitter_bot.core$retweet_eig.invoke(core.clj:43)
        at eigforever_twitter_bot.core$_main.invokeStatic(core.clj:52)
        at eigforever_twitter_bot.core$_main.invoke(core.clj:50)
        at clojure.lang.AFn.applyToHelper(AFn.java:152)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at eigforever_twitter_bot.core.main(Unknown Source)

[chbrown]

That's a terrible traceback and completely unhelpful error message, I'll admit, but it has nothing to do with aleph, Ring, or docker containerization :) Blame clj-oauth? Or just Java? 😝 That's what I do (because I've shot myself in the foot a couple times with that same NPE).

You're getting that error because your UserCredentials aren't filled in. See https://github.com/chbrown/twttr/blob/master/src/twttr/auth.clj#L75-L76

Imma leave this ticket open for myself, though, as a reminder to add some asserts or something.

[bzg]

Thanks a lot for the feedback. Indeed, I was passing environment variables through the docker run command line, but forgot to add the relevant ENV entries in the Dockerfile before.

I just updated https://github.com/bzg/eigforever-twitter-bot to reflect the version I'm using. With this version, (println (creds)) within the -main function will correctly print the credentials.

Still, there seem to be an authorization issue with the dockerized version that I don't have with the standalone jar file. Here is the error log (sorry to dump so many lines here):

<h2>HTTP ERROR: 401</h2>
<p>Problem accessing '/1.1/statuses/filter.json'. Reason:
<pre>    Unauthorized</pre>
</body>
</html> {:aleph/keep-alive? true, :aleph/complete << false >>,
:headers {"server" "tsa_b", "content-type" "text/html",
"x-tsa-request-body-time" "0", "x-connection-hash"
"98e9ee0414a4a5a1620764d1c996fa78", "strict-transport-security"
"max-age=631138519", "www-authenticate" "OAuth realm=\"Firehose\"",
"x-response-time" "9", "transfer-encoding" "chunked", "set-cookie"
"personalization_id=\"v1_T4W/lvkwCADF/8HLTgthDQ==\"; Max-Age=63072000;
Expires=Thu, 4 Mar 2021 08:22:57 GMT; Path=/;
Domain=.twitter.com,guest_id=v1%3A155177417704806054;
Max-Age=63072000; Expires=Thu, 4 Mar 2021 08:22:57 GMT; Path=/;
Domain=.twitter.com", "date" "Tue, 05 Mar 2019 08:22:57 GMT",
"x-xss-protection" "1; mode=block;
report=https://twitter.com/i/xss_report", "cache-control"
"must-revalidate,no-cache,no-store"}, :status 401, :body
"<html>\\n<head>\\n<meta http-equiv=\"Content-Type\"
content=\"text/html; charset=utf-8\"/>\\n<title>Error 401
Unauthorized</title>\n</head>\n<body>\n<h2>HTTP ERROR:
401</h2>\n<p>Problem accessing
'/1.1/statuses/filter.json'. Reason:\n<pre>
Unauthorized</pre>\n</body>\n</html>"}
        at twttr.api$ex_twitter.invokeStatic(api.clj:44)
        at twttr.api$ex_twitter.invoke(api.clj:37)
        at twttr.api$request_endpoint$fn__6993.invoke(api.clj:106)
        at manifold.deferred$catch_SINGLEQUOTE_$fn__1279.invoke(deferred.clj:963)
        at manifold.deferred.Listener.onError(deferred.clj:220)
        at manifold.deferred.Deferred$fn__1055.invoke(deferred.clj:399)
        at manifold.deferred.Deferred.error(deferred.clj:399)
        at manifold.deferred$error_BANG_.invokeStatic(deferred.clj:250)
        at manifold.deferred$error_BANG_.invoke(deferred.clj:247)
        at manifold.deferred$fn__1185$subscribe__1186$fn__1189.invoke(deferred.clj:711)
        at manifold.deferred.Listener.onError(deferred.clj:220)
        at manifold.deferred.Deferred$fn__1055.invoke(deferred.clj:399)
        at manifold.deferred.Deferred.error(deferred.clj:399)
        at manifold.deferred$error_BANG_.invokeStatic(deferred.clj:250)
        at manifold.deferred$error_BANG_.invoke(deferred.clj:247)
        at manifold.deferred$fn__1228$subscribe__1229$fn__1232.invoke(deferred.clj:805)
        at manifold.deferred.Listener.onError(deferred.clj:220)
        at manifold.deferred.Deferred$fn__1055$fn__1056.invoke(deferred.clj:399)
        at clojure.lang.AFn.run(AFn.java:22)
        at io.aleph.dirigiste.Executor$3.run(Executor.java:318)
        at io.aleph.dirigiste.Executor$Worker$1.run(Executor.java:62)
        at manifold.executor$thread_factory$reify__556$f__557.invoke(executor.clj:44)
        at clojure.lang.AFn.run(AFn.java:22)
        at java.lang.Thread.run(Thread.java:748)

Could it be somehow related to the 0.0.0.0 issue first mentioned?

[chbrown]

I don't know why you'd see that on a VM vs. barebones, especially since it's clearly an authentication issue. Are you sure everything is the same as if you ran it directly (not on Docker)?

E.g., if you have already made a connection to the streaming API locally, and you start a docker container using the same credentials, that is not like the original stream, because Twitter does not allow more than one connection to the streaming API per set of authentication credentials. Rather, to compare dockerized version vs. standalone jar, you would start a second process locally, using the same credentials, also connecting to the streaming API.

N.b.: Twitter's idea of whether your credentials are free to connect to a new streaming API endpoint is not necessarily what it looks like to you. Sometimes it takes several minutes for them to realize your connection no longer exists, after you have killed it locally.

If this were related to the "0.0.0.0 issue" (?), I'm like 99.9% sure the fix would involve some additional configuration of docker's network / bridge / etc. settings, rather than any changes to twttr. Unfortunately, I know very little about docker configuration 😕

[bzg]

Hi, I've checked again and this was a mistake in the way I called the environment variables from Docker. So unless you want to "Throw a more informative exception on missing credentials", the issue can be closed.

Thanks for your feedback so far!