search

pixelated / pixelated-user-agent

User facing components of Pixelated: a JavaScript single page app and a RESTful service.
GNU Affero General Public License v3.0
159 stars 72 forks source link

Soledad streaming branch merge #888

Closed anikarni closed 7 years ago

anikarni commented 7 years ago

This issue was created to test and merge the new streaming branch from leap. This does not include migrations, as the changes are backward compatible. There will be another issue for migrations.

anikarni commented 7 years ago

Blocked by issue: https://leap.se/code/issues/8732

anikarni commented 7 years ago

The streaming changes changed the contract for the Soledad constructer, so we had to change how we called Soledad. There is a commit in branch soledad-merge for that.

Moreover, to test, we used the libraries directly from leap in the service/requirements.txt. We recommend only using the develop branch, because master changes the version and might make it impossible to revert. If you need to roll back the version for Soledad, there is documentation on how to revert with reprepro in the wiki.

You can then package the new soledad version for debian with the pipeline, changing the material for the soledad-jessie pipeline to the leap/develop branch. You can use that to deploy to a test environment. The same goes for testing the soledad-merge branch in Pixelated.

deniscostadsc commented 7 years ago

We tested just Soledad streaming branch and still having errors. We sent an email to drebs and shyba. We are waiting for the response.

varac commented 7 years ago

@thaissiqueira @tayanefernandes regarding the pixelated-user-agent tls cert problem on unstable.pix:

When you install/upgrade the pixelated-user-agent package, it will replace its config file /etc/default/pixelated-server. The default TLS values for that config are:

LEAP_PROVIDER=localhost
SSL_KEY=/etc/ssl/private/ssl-cert-snakeoil.key
SSL_CERT=/etc/ssl/certs/ssl-cert-snakeoil.pem

This will not work out of the box. Therefore we configure it with puppet-pixelated to use the proper cert values:

LEAP_PROVIDER=unstable.pixelated-project.org
SSL_KEY=/etc/x509/keys/leap_commercial.key
SSL_CERT=/etc/x509/certs/leap_commercial.crt

If you don't want to deploy after a manual upgrade of the package, you can manually update that file.

Or, because we are storing /etc in git using etckeeper you can simply check out an older version, i.e.:

cd /etc/default/
# find a sane config version
git log pixelated-server
git checkout 5aaae92c3b17d3b1a0813df930a0680606fc4bc4 pixelated-server

After that, restarting pixelated-server works again.

tayanefernandes commented 7 years ago

After we installed soledad 0.8.1+nmu2~319.gbp193573 and pixelated 0.6~582.gbpf21558, ran the tests and had success on that, during the day, we got this error on staging:

C Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: 2017-02-21 21:18:06 [twisted.internet.defer] CRITICAL Unhandled error in Deferred:
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: 2017-02-21 21:18:06 [twisted.internet.defer] CRITICAL
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: Traceback (most recent call last):
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: File "/usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1184, in gotResult
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: _inlineCallbacks(r, g, deferred)
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: File "/usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1174, in _inlineCallbacks
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: deferred.errback()
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: File "/usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 434, in errback
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: self._startRunCallbacks(fail)
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: File "/usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 501, in _startRunCallbacks
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: self._runCallbacks()
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: --- <exception caught here> ---
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: File "/usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 588, in _runCallbacks
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: current.result = callback(current.result, *args, **kw)
C Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: File "/usr/share/python/pixelated-user-agent/src/leap.soledad.client/client/src/leap/soledad/client/http_target/fetch_protocol.py", line 132, in _error
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: self.transport.loseConnection()
C Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: exceptions.AttributeError: 'TransportProxyProducer' object has no attribute 'loseConnection'
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: 2017-02-21 21:18:06 [leap.soledad.client.http_target.fetch_protocol] ERROR [Failure instance: Traceback: <class 'leap.soledad.client._crypto.InvalidBlob'>: Invalid Tag. Blob authentication failed.
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:434:errback
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:501:_startRunCallbacks
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:588:_runCallbacks
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:1184:gotResult
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: --- <exception caught here> ---
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:1126:_inlineCallbacks
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/python/failure.py:389:throwExceptionIntoGenerator
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: /usr/share/python/pixelated-user-agent/src/leap.soledad.client/client/src/leap/soledad/client/http_target/fetch.py:112:_doc_parser
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:1126:_inlineCallbacks
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/python/failure.py:389:throwExceptionIntoGenerator
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: /usr/share/python/pixelated-user-agent/src/leap.soledad.client/client/src/leap/soledad/client/http_target/fetch.py:118:__atomic_doc_parse
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:588:_runCallbacks
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: /usr/share/python/pixelated-user-agent/src/leap.soledad.client/client/src/leap/soledad/client/_crypto.py:324:<lambda>
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: /usr/share/python/pixelated-user-agent/src/leap.soledad.client/client/src/leap/soledad/client/_crypto.py:313:_end_stream
. Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: ]

On dev and unstable we got this error:

[leap.soledad.client.api] ERROR got exception when syncing!
Failure: twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused.

We suspect that this error can be related with a connection with l2db.

@kalikaneko @shyba and drebs Do you have any ideia about what could be happening?

We are not comfortable in send this code to production because of those errors.

kalikaneko commented 7 years ago

After we installed soledad 0.8.1+nmu2~319.gbp193573

the version seems weird, at least. where is this deb coming from? which commit from master is that? can you report what is there for a GET at the root of the soledad server?

and pixelated 0.6~582.gbpf21558, ran the tests and had success on that, during the day, we got this error on staging:

C Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: 2017-02-21 21:18:06 [twisted.internet.defer] CRITICAL Unhandled error in Deferred: . Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: 2017-02-21 21:18:06 [twisted.internet.defer] CRITICAL . Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: Traceback (most recent call last): . Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: File "/usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1184, in gotResult . Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: _inlineCallbacks(r, g, deferred) . Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: File "/usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1174, in _inlineCallbacks . Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: deferred.errback() . Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: File "/usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 434, in errback . Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: self._startRunCallbacks(fail) . Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: File "/usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 501, in _startRunCallbacks . Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: self._runCallbacks() . Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: --- --- . Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: File "/usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 588, in _runCallbacks . Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: current.result = callback(current.result, *args, **kw) C Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: File "/usr/share/python/pixelated-user-agent/src/leap.soledad.client/client/src/leap/soledad/client/http_target/fetch_protocol.py", line 132, in _error . Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: self.transport.loseConnection() C Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: exceptions.AttributeError: 'TransportProxyProducer' object has no attribute 'loseConnection' . Feb 21 21:18:06 pixelated pixelated-user-agent[13705]: 2017-02-21 21:18:06 [leap.soledad.client.http_target.fetch_protocol] ERROR [Failure instance: Traceback: <class 'leap.soledad.client._crypto.InvalidBlob'>: Invalid Tag. Blob authentication failed.

Looking at it, I think this means either corrupted download, or the stream being interrupted (which would trigger a call to _end_stream in BlobDecryptor).

Even though this is not covered by the tests, I think it should be possible to reproduce the error easily by triggering one of those two conditions. That should be the precondition to fix this.

(thinking about a solution, the newly introduced size in the preamble should help catching the second case. for the first one, I think we can only discard and retry, unless there is actually data tampering in which I really don't know what's the sensible way to proceed).

On dev and unstable we got this error:

[leap.soledad.client.api] ERROR got exception when syncing! Failure: twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused.

this looks to me like the remote soledad server went down. maybe a restart of the server?

I think this one should be recovered gracefully.

-- We reject kings, presidents and voting. We believe in rough consensus and working code.

tayanefernandes commented 7 years ago

Hey @kalikaneko, thanks for replying! ☺️

where is this deb coming from? which commit from master is that?

The commit is the last from master branch (193573cf), built on our pipeline. The tags looks weird, because we were not updating it.

can you report what is there for a GET at the root of the soledad server?

{"version": "0.8.1+276.g193573c.dirty", "blobs": false} What also looks weird, because our debian package has the pipeline iteration number 319 and this version information has 276. Should they be the same?

On staging environment:

Even though this is not covered by the tests, I think it should be possible to reproduce the error easily by triggering one of those two conditions.

We're not sure how to reproduce those conditions, but we're not getting those exceptions right now. We're getting this one:

Feb 22 21:40:24 pixelated pixelated-user-agent[13731]: 2017-02-22 21:40:24 [leap.bitmask.mail.incoming.service] ERROR [Failure instance: Traceback (failure with no frames): <class 'leap.soledad.common.l2db.errors.HTTPError'>: HTTPError(401, 'Unauthorized')

On dev and unstable environments:

this looks to me like the remote soledad server went down. maybe a restart of the server?

We restarted a couple of times and the state is persisting.

varac commented 7 years ago

@tayanefernandes @kalikaneko

{"version": "0.8.1+276.g193573c.dirty", "blobs": false} What also looks weird, because our debian package has the pipeline iteration number 319 and this version information has 276. Should they be the same?

The most important information imo is the commit id the package is build from, which is the one after the semantic version, prefixed with a 'g', so 193573c. This refers to the current pixelated:master HEAD:

--- git/soledad ‹pixelated_master* ⁇› » git show | cat
commit 193573cf8d44a3b6a7d8ae0e43988cffb38a428a (HEAD -> pixelated_master, pixelated/master, origin/master, origin/HEAD, master)
Author: Kali Kaneko <kali@leap.se>
Date:   Fri Feb 17 16:52:07 2017

    [bug] reuse wsgi threadpool
...

Our soledad build pipeline just increments the number of BUILDS (319) and is different to the number of git commits after the last tag (276).

varac commented 7 years ago

On staging:

These unauth errors in the useragent.log:

Feb 23 14:33:36 pixelated pixelated-user-agent[13731]: Failure: leap.soledad.common.l2db.errors.HTTPError: HTTPError(401, 'Unauthorized')
Feb 23 14:33:36 pixelated pixelated-user-agent[13731]: 2017-02-23 14:33:36 [leap.bitmask.mail.incoming.service] ERROR [Failure instance: Traceback (failure with no frames): <class 'leap.soledad.common.l2db.errors.HTTPError'>: HTTPError(401, 'Unauthorized')

related to these soledad-server errors in the syslog:

Feb 23 14:33:36 pixelated soledad-server: [-] 0.0.0.0 - - [23/Feb/2017:13:33:35 +0000] "GET /user-c5f22e56c1b981f32abeae3a601f1027/sync-from/48c10ce365c249ca96365ffaa91029c4 HTTP/1.1" 401 12 "-" "-"
varac commented 7 years ago

@kalikaneko we noticed that a restart of pixelated-user-agent fixed the unauth error for now, we're rebuilding staging.pix to see if it's working better now. Please hold on until we investigated.

tuliocasagrande commented 7 years ago

hey @kalikaneko @varac, we ran staging a couple of hours without significant errors, but at the end of day, we started getting the TransportProxyProducer error again.

On dev and unstable, we had to manually restart soledad-server service after deploying.

We are seeing old and new emails, but also getting different random errors, like this one:

Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: 2017-02-23 21:23:20 [leap.bitmask.mail.incoming.service] ERROR [Failure instance: Traceback: <class 'leap.bitmask.keymanager.errors.DecryptError'>: Failed to encrypt/decrypt: [GNUPG:] ENC_TO 86546EB979C6A6D0 1 0
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: gpg: encrypted with RSA key, ID 79C6A6D0
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: [GNUPG:] NO_SECKEY 86546EB979C6A6D0
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: [GNUPG:] BEGIN_DECRYPTION
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: [GNUPG:] DECRYPTION_FAILED
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: gpg: decryption failed: secret key not available
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: [GNUPG:] END_DECRYPTION
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:1126:_inlineCallbacks
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/python/failure.py:389:throwExceptionIntoGenerator
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: /usr/share/python/pixelated-user-agent/src/leap.bitmask/src/leap/bitmask/keymanager/__init__.py:591:decrypt_with_inactive_keys
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:1274:unwindGenerator
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: --- <exception caught here> ---
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:1128:_inlineCallbacks
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: /usr/share/python/pixelated-user-agent/src/leap.bitmask/src/leap/bitmask/keymanager/__init__.py:583:decrypt_with_inactive_keys
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: ]
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: 2017-02-23 21:23:20 [leap.bitmask.mail.incoming.service] INFO processing decrypted doc
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: 2017-02-23 21:23:20 [leap.bitmask.mail.incoming.service] ERROR [Failure instance: Traceback: <type 'exceptions.ValueError'>: No JSON object could be decoded
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:1184:gotResult
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:1174:_inlineCallbacks
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:434:errback
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:501:_startRunCallbacks
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: --- <exception caught here> ---
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:588:_runCallbacks
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: /usr/share/python/pixelated-user-agent/src/leap.bitmask/src/leap/bitmask/mail/incoming/service.py:338:process_decrypted
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: /usr/share/python/pixelated-user-agent/src/leap.bitmask/src/leap/bitmask/mail/incoming/service.py:365:_process_decrypted_doc
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: /usr/share/python/pixelated-user-agent/src/leap.bitmask/src/leap/bitmask/mail/utils.py:352:json_loads
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: /usr/lib/python2.7/json/__init__.py:351:loads
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: /usr/lib/python2.7/json/decoder.py:366:decode
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: /usr/lib/python2.7/json/decoder.py:384:raw_decode
Feb 23 21:23:20 unstable1 pixelated-user-agent[4121]: ]
tuliocasagrande commented 7 years ago

I have just figured out that this 'TransportProxyProducer' object has no attribute 'loseConnection' is a side effect when trying to close the connection of a first actual error:

fetch_protocol.py#L130

def _error(self, reason):
        logger.error(reason)
        self.transport.loseConnection()

Since the original error is already being logged, I'd say it should be tackled first.

tuliocasagrande commented 7 years ago

A new error we've got when running locally against the unstable environment.

We debugged a little bit and noticed self._factory is the user's uuid.

2017-03-01 19:16:26 [pixelated.resources.login_resource] ERROR Login error during thais services setup: 'str' object is not callable
 Traceback (most recent call last):
  File "/home/vagrant/.venvs/pixua/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 434, in errback
    self._startRunCallbacks(fail)
  File "/home/vagrant/.venvs/pixua/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 501, in _startRunCallbacks
    self._runCallbacks()
  File "/home/vagrant/.venvs/pixua/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 588, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/vagrant/.venvs/pixua/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1184, in gotResult
    _inlineCallbacks(r, g, deferred)
--- <exception caught here> ---
  File "/home/vagrant/.venvs/pixua/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/vagrant/.venvs/pixua/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/vagrant/service/pixelated/config/leap.py", line 95, in setup
    leap_session = yield create_leap_session(self._provider, user_auth.username, password, user_auth)
  File "/home/vagrant/.venvs/pixua/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/vagrant/.venvs/pixua/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/vagrant/service/pixelated/config/leap.py", line 57, in create_leap_session
    leap_session = yield LeapSessionFactory(provider).create(username, password, auth)
  File "/home/vagrant/.venvs/pixua/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/vagrant/.venvs/pixua/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/vagrant/service/pixelated/config/sessions.py", line 53, in create
    session = yield self._create_new_session(username, password, auth)
  File "/home/vagrant/.venvs/pixua/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/vagrant/.venvs/pixua/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/vagrant/service/pixelated/config/sessions.py", line 66, in _create_new_session
    soledad = yield self.setup_soledad(auth.token, auth.uuid, password, api_cert)
  File "/home/vagrant/.venvs/pixua/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/vagrant/.venvs/pixua/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/vagrant/service/pixelated/config/sessions.py", line 98, in setup_soledad
    auth_token=user_token)
  File "/home/vagrant/.venvs/pixua/local/lib/python2.7/site-packages/twisted/python/threadpool.py", line 246, in inContext
    result = inContext.theWork()
  File "/home/vagrant/.venvs/pixua/local/lib/python2.7/site-packages/twisted/python/threadpool.py", line 262, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/home/vagrant/.venvs/pixua/local/lib/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/home/vagrant/.venvs/pixua/local/lib/python2.7/site-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/home/vagrant/.venvs/pixua/src/leap.soledad.client/client/src/leap/soledad/client/api.py", line 198, in __init__
    self._secrets = Secrets(self)
  File "/home/vagrant/.venvs/pixua/src/leap.soledad.client/client/src/leap/soledad/client/_secrets/__init__.py", line 44, in __init__
    self._bootstrap()
  File "/home/vagrant/.venvs/pixua/src/leap.soledad.client/client/src/leap/soledad/client/_secrets/__init__.py", line 64, in _bootstrap
    encrypted = self.storage.load_remote()
  File "/home/vagrant/.venvs/pixua/src/leap.soledad.client/client/src/leap/soledad/client/_secrets/storage.py", line 106, in load_remote
    doc = self._remote_doc
  File "/home/vagrant/.venvs/pixua/src/leap.soledad.client/client/src/leap/soledad/client/_secrets/storage.py", line 91, in _remote_doc
    doc = self._get_remote_doc()
  File "/home/vagrant/.venvs/pixua/src/leap.soledad.client/client/src/leap/soledad/client/_secrets/util.py", line 54, in _decorated
    result = method(self, *args, **kwargs)
  File "/home/vagrant/.venvs/pixua/src/leap.soledad.client/client/src/leap/soledad/client/_secrets/storage.py", line 98, in _get_remote_doc
    doc = self._shared_db.get_doc(self._remote_doc_id())
  File "/home/vagrant/.venvs/pixua/src/leap.soledad.common/common/src/leap/soledad/common/l2db/remote/http_database.py", line 103, in get_doc
    doc = self._factory(doc_id, doc_rev, res)
exceptions.TypeError: 'str' object is not callable
kalikaneko commented 7 years ago

So, there are different issues piling up here. We should link to the separate soledad issues, and make sure that those are given high priority in our side.

kalikaneko commented 7 years ago

https://0xacab.org/leap/soledad/issues/8783 is the "TypeError: 'str' object is not callable"

tuliocasagrande commented 7 years ago

@kalikaneko

We have deployed 32c6d2ea on dev, unstable and staging environments.

Staging is running smoothly and without any errors! But keep in mind staging is always deployed with zero accounts, so every user is brand new (no backward compatibility is required).

On dev and unstable, we are testing with multiple old users and while some of them are getting no errors, others are getting the Blob authentication failed.

Before opening a specific issue for that, we noticed the problematic users are the ones we have logged in before yesterday's fix. So we wanted to clean (delete) client side's leap folder to remove any inconsistent state. As far as I remember, the server side is supposed to re-create it, right?

@varac Do you know where is the leap folder on our side?

varac commented 7 years ago

@tuliocasagrande: /var/lib/pixelated/.leap/

deniscostadsc commented 7 years ago

It was merged. 🎉

Remaining issues: https://0xacab.org/leap/soledad/issues/8786 https://0xacab.org/leap/soledad/issues/8787 https://0xacab.org/leap/soledad/issues/8788

1003