search

Evidlo / passhole

A secure hole for your passwords (KeePass CLI)
GNU General Public License v3.0
199 stars 19 forks source link

Destructive write with edit #26

Closed chew-z closed 4 years ago

chew-z commented 5 years ago

I have been using passhole for some time now, very happy with it. Recently I have changed couple of passwords through ph edit command. Unfortunately after closing the session the database cannot be read with following error (in essence).

'utf-8' codec can't decode byte 0xd0 in position 0: invalid continuation byte

Now of course I had a backup, so no harm done. I have tried couple more times with identical result (database cannot be read by passhole after changing password and closing session). I have also tried with my own simple python script that is using libkeepass for printing database content and it had failed with identical error.

So my guess is that ph edit is making destructive writes of the kdbx database (probably wrong handling of unicode characters).

I am using version 1.9.post1 but I have also tried going back to 1.5 - with identical error.

Full error listing:

Traceback (most recent call last):
  File "/home/rrj/.local/bin/ph", line 10, in <module>
    sys.exit(main())
  File "/home/rrj/.local/lib/python3.7/site-packages/passhole/passhole.py", line 927, in main
    args.func(args)
  File "/home/rrj/.local/lib/python3.7/site-packages/passhole/passhole.py", line 489, in show
    databases = open_databases(**vars(args))
  File "/home/rrj/.local/lib/python3.7/site-packages/passhole/passhole.py", line 391, in open_databases
    s.get('gpgkey')
  File "/home/rrj/.local/lib/python3.7/site-packages/passhole/passhole.py", line 354, in open_database
    return PyKeePass(database, password=password, keyfile=keyfile)
  File "/home/rrj/.local/lib/python3.7/site-packages/pykeepass_cache/pykeepass_cache.py", line 176, in PyKeePass
    return _fork_and_run(func, timeout=timeout, socket_path=socket_path)
  File "/home/rrj/.local/lib/python3.7/site-packages/pykeepass_cache/pykeepass_cache.py", line 95, in _fork_and_run
    return func(conn)
  File "/home/rrj/.local/lib/python3.7/site-packages/pykeepass_cache/pykeepass_cache.py", line 175, in <lambda>
    func = lambda conn: conn.root.PyKeePass(filename, password, keyfile, transformed_key)
  File "/home/rrj/.local/lib/python3.7/site-packages/rpyc/core/netref.py", line 247, in __call__
    return syncreq(_self, consts.HANDLE_CALL, args, kwargs)
  File "/home/rrj/.local/lib/python3.7/site-packages/rpyc/core/netref.py", line 76, in syncreq
    return conn.sync_request(handler, proxy, *args)
  File "/home/rrj/.local/lib/python3.7/site-packages/rpyc/core/protocol.py", line 464, in sync_request
    return self.async_request(handler, *args, timeout=timeout).value
  File "/home/rrj/.local/lib/python3.7/site-packages/rpyc/core/async_.py", line 102, in value
    raise self._obj
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 0: invalid continuation byte

========= Remote Traceback (1) =========
Traceback (most recent call last):
  File "/home/rrj/.local/lib/python3.7/site-packages/rpyc/core/protocol.py", line 323, in _dispatch_request
    res = self._HANDLERS[handler](self, *args)
  File "/home/rrj/.local/lib/python3.7/site-packages/rpyc/core/protocol.py", line 585, in _handle_call
    return obj(*args, **dict(kwargs))
File "/home/rrj/.local/lib/python3.7/site-packages/pykeepass_cache/pykeepass_cache.py", line 43, in exposed_PyKeePass
    kp = PyKeePass(filename, password, keyfile)
  File "/home/rrj/.local/lib/python3.7/site-packages/pykeepass/pykeepass.py", line 42, in __init__
    transformed_key=transformed_key
  File "/home/rrj/.local/lib/python3.7/site-packages/pykeepass/pykeepass.py", line 62, in read
    transformed_key=transformed_key
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 325, in parse_file
    return self.parse_stream(f, **contextkw)
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 316, in parse_stream
    return self._parsereport(stream, context, "(parsing)")
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 328, in _parsereport
    obj = self._parse(stream, context, path)
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 1979, in _parse
    subobj = sc._parsereport(stream, context, path)
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 328, in _parsereport
    obj = self._parse(stream, context, path)
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 2468, in _parse
    return self.subcon._parsereport(stream, context, path)
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 328, in _parsereport
    obj = self._parse(stream, context, path)
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 3663, in _parse
    return sc._parsereport(stream, context, path)
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 328, in _parsereport
    obj = self._parse(stream, context, path)
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 1979, in _parse
    subobj = sc._parsereport(stream, context, path)
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 328, in _parsereport
    obj = self._parse(stream, context, path)
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 2468, in _parse
    return self.subcon._parsereport(stream, context, path)
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 328, in _parsereport
    obj = self._parse(stream, context, path)
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 715, in _parse
    return self._decode(obj, context, path)
  File "/home/rrj/.local/lib/python3.7/site-packages/pykeepass/kdbx_parsing/common.py", line 70, in _decode
    return subcon_out.parse(data, **con)
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 304, in parse
    return self.parse_stream(io.BytesIO(data), **contextkw)
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 316, in parse_stream
    return self._parsereport(stream, context, "(parsing)")
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 328, in _parsereport
    obj = self._parse(stream, context, path)
File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 2468, in _parse
    return self.subcon._parsereport(stream, context, path)
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 328, in _parsereport
    obj = self._parse(stream, context, path)
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 715, in _parse
    return self._decode(obj, context, path)
  File "/home/rrj/.local/lib/python3.7/site-packages/pykeepass/kdbx_parsing/common.py", line 70, in _decode
    return subcon_out.parse(data, **con)
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 304, in parse
    return self.parse_stream(io.BytesIO(data), **contextkw)
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 316, in parse_stream
    return self._parsereport(stream, context, "(parsing)")
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 328, in _parsereport
    obj = self._parse(stream, context, path)
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 1979, in _parse
    subobj = sc._parsereport(stream, context, path)
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 328, in _parsereport
    obj = self._parse(stream, context, path)
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 2468, in _parse
    return self.subcon._parsereport(stream, context, path)
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 328, in _parsereport
    obj = self._parse(stream, context, path)
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 3663, in _parse
    return sc._parsereport(stream, context, path)
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 328, in _parsereport
    obj = self._parse(stream, context, path)
  File "/home/rrj/.local/lib/python3.7/site-packages/construct/core.py", line 715, in _parse
    return self._decode(obj, context, path)
  File "/home/rrj/.local/lib/python3.7/site-packages/pykeepass/kdbx_parsing/common.py", line 189, in _decode
    ).decode('utf-8') if unicodedata.category(c)[0] != "C")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 0: invalid continuation byte
Evidlo commented 5 years ago

Can you more detail for what edits you are making? Unicode inputs seem to work fine on my end:

``` [evan@blackbox tmp] ph --database test3.kdbx --keyfile test3.key ls Enter password: foobar_entry quote test -> " <- root_entry testing_new foobar_group ├── foobar_entry ├── group_entry └── subgroup ├── foobar_entry ├── subentry ├── subentry2 └── subgroup2 foobar_group2 Работа └── Тест [evan@blackbox tmp] ph --database test3.kdbx --keyfile test3.key edit foobar_group/foobar_entry Enter password: Notes: ° Password: foobar° Title: foobar_entry° URL: ° UserName: foobar° [evan@blackbox tmp] ph --database test3.kdbx --keyfile test3.key ls Enter password: foobar_entry quote test -> " <- root_entry testing_new foobar_group ├── foobar_entry° ├── group_entry └── subgroup ├── foobar_entry ├── subentry ├── subentry2 └── subgroup2 foobar_group2 Работа └── Тест ```
chew-z commented 5 years ago

I had been attempting to change password for three of my accounts. After 1st failure (with some fancy chars in generated passwords) I have got rid of special characters in passwords but it didn't help.

ph edit 'Shopping/Allegro'

Tell me how I could be of help identifying the reasons for the error - I would like very much to have passhole working for me again ;-)

chew-z commented 5 years ago

It seems like it is completly borked.

Fresh empty database right after ph init is failing on ph listwith:

ph list                 
Enter password (passhole): 
Traceback (most recent call last):
  File "/home/rrj/.local/bin/ph", line 10, in <module>
    sys.exit(main())
  File "/home/rrj/.local/lib/python3.7/site-packages/passhole/passhole.py", line 927, in main
    args.func(args)
  File "/home/rrj/.local/lib/python3.7/site-packages/passhole/passhole.py", line 518, in list_entries
    databases = open_databases(**vars(args))
  File "/home/rrj/.local/lib/python3.7/site-packages/passhole/passhole.py", line 391, in open_databases
    s.get('gpgkey')
  File "/home/rrj/.local/lib/python3.7/site-packages/passhole/passhole.py", line 354, in open_database
    return PyKeePass(database, password=password, keyfile=keyfile)
  File "/home/rrj/.local/lib/python3.7/site-packages/pykeepass_cache/pykeepass_cache.py", line 176, in PyKeePass
    return _fork_and_run(func, timeout=timeout, socket_path=socket_path)
  File "/home/rrj/.local/lib/python3.7/site-packages/pykeepass_cache/pykeepass_cache.py", line 95, in _fork_and_run
    return func(conn)
  File "/home/rrj/.local/lib/python3.7/site-packages/pykeepass_cache/pykeepass_cache.py", line 175, in <lambda>
    func = lambda conn: conn.root.PyKeePass(filename, password, keyfile, transformed_key)
  File "/home/rrj/.local/lib/python3.7/site-packages/rpyc/core/netref.py", line 247, in __call__
    return syncreq(_self, consts.HANDLE_CALL, args, kwargs)
  File "/home/rrj/.local/lib/python3.7/site-packages/rpyc/core/netref.py", line 76, in syncreq
    return conn.sync_request(handler, proxy, *args)
  File "/home/rrj/.local/lib/python3.7/site-packages/rpyc/core/protocol.py", line 464, in sync_request
    return self.async_request(handler, *args, timeout=timeout).value
  File "/home/rrj/.local/lib/python3.7/site-packages/rpyc/core/async_.py", line 102, in value
    raise self._obj
rpyc.core.vinegar/construct.core.ChecksumError: wrong checksum, read b'f7abc2d49e03f5062555765bb208d222f505c5f75e5e9bdd95dcd5e98e3866fa', computed b'13125a090663fb43ebc83c864156f7024e5db88e935f4596f03a489503e68345'
Evidlo commented 5 years ago

Try rolling back to commit 1bbf0, which is before I added database caching via rpyc. This shouldn't affect your first error though.

Evidlo commented 5 years ago

Can you replicate this in the Docker environment? make docker_debian will put you in a fresh environment with passhole installed. I haven't been able to reproduce this:

[evan@blackbox passhole] make docker_debian 
............
root@8ddbda849f45:/home/passhole# ph --version
1.9.post1
root@8ddbda849f45:/home/passhole# ph init
Database name (no spaces): passhole
Desired database path: /root/.local/passhole/passhole.kdbx
Password protect database? (Y/n): n
Use a keyfile? (Y/n): n
Creating database at /root/.local/passhole/passhole.kdbx
Creating config at /root/.config/passhole.ini
root@8ddbda849f45:/home/passhole# ph ls
root@8ddbda849f45:/home/passhole# ph add lkj
Username: foo
Password: 
Confirm: 
URL: foo
root@8ddbda849f45:/home/passhole# ph ls
lkj
root@8ddbda849f45:/home/passhole# ph edit lkj
Title: lkj°
UserName: foo°
Password: foo
URL: foo
root@8ddbda849f45:/home/passhole# ph ls
lkj°

I may add an option to automatically create a backup upon successful database decryption to help mitigate any data loss.

chew-z commented 5 years ago

I am sorry but I cannot use Docker as my CLI is within chroot environment.

I have stopped using passhole temporarily. I will try with some new version in a few weeks.

I have also noticed that all/most of password fields are ignored by other Keepass app (Keepass Touch on iOS) however ther are visible in older versions of my kdbx (before ph edit). Weird by I think this might be result of my awkward environment (Secure Shell, etc.).

As for backup I think that anyway everybody needs some backup and sync strategy of its own so I am not sure if it is required feature for passhole.

Evidlo commented 4 years ago

Finally fixed this with pykeepass 3.2.0. Can you verify that the issue is gone for you?

edit: what I fixed might be unrelated, so I'm interested in seeing if you have the problem still. One thing you might try is the --no-cache option, which disables background caching of the database.

chew-z commented 4 years ago

Sure. I will test it. This week. Just need little time.

chew-z commented 4 years ago

Sorry it took so long. But I have tested today and it works! ;-)

I have tested in both Crostini and Crosh (I was afraid that Crosh is doing something smart with endcoding) and it works on both. If I edit password field it stays that way even after reboot and not busting the database.

Thanks a lot. Passhole is real savior.

I have to to clean up my Keepass database from cruft so I will be doing a lot of editing today. Will let you know if I have some issues.