zodbconvert: export from postrges to filestorage with blobs?

[agitator]

Is there a way to export/backup a postgres/relstorage (shared blobs off) to local filestorage with blobs for development?

Prior to version 3, I could backup the db to local filestorage including blobs with something like this

<relstorage source>
        # ZODB Cache Dir
        # blob-dir /blobstorage
        # db connect
        <postgresql>
        dsn dbname='db' user='user' host='db' password='...'
        </postgresql>
</relstorage>

<filestorage destination>
        #zeo data
        path ./backup/Data.fs
        #blobs
        # blob-dir ./backup/blobstorage
</filestorage>

Background... I'm using a managed postgres from Google cloud and fighting their own way of import/exporting data, running in to special user/roles constraints...

[jamadden]

As far as I know, nothing has changed in that regard. You should be able to move blobs around, provided that both storages support blobs (in your snippet, they don't). I'm pretty sure there are explicit test cases for exactly this scenario.

Are you getting some sort of error?

[agitator]

Config:

<filestorage destination>
        path ./backup/Data.fs
        blob-dir ./backup/blobstorage
</filestorage>

Traceback:

❯ ./bin/zodbconvert backup-from-relstorage.cfg
2020-12-02 12:50:04,529 [ZODB.blob] INFO (70184) Blob directory ./backup/blobstorage does not exist. Selected `bushy` layout. 
2020-12-02 12:50:04,529 [ZODB.blob] INFO (70184) Blob directory '/Users/peter/workspace/project/buildout/backup/blobstorage/' does not exist. Created new directory.
2020-12-02 12:50:04,529 [ZODB.blob] INFO (70184) Blob temporary directory './backup/blobstorage/tmp' does not exist. Created new directory.
2020-12-02 12:50:04,530 [zodbconvert] INFO Storages opened successfully.
Traceback (most recent call last):
  File "./bin/zodbconvert", line 320, in <module>
    sys.exit(relstorage.zodbconvert.main())
  File "/Users/peter/workspace/project/buildout/eggs/RelStorage-3.3.2-py3.7-macosx-10.15-x86_64.egg/relstorage/zodbconvert.py", line 178, in main
    destination.copyTransactionsFrom(source)
  File "/Users/peter/workspace/project/buildout/eggs/ZODB-5.6.0-py3.7.egg/ZODB/FileStorage/FileStorage.py", line 342, in copyTransactionsFrom
    return BlobStorageMixin.copyTransactionsFrom(self, other)
  File "/Users/peter/workspace/project/buildout/eggs/ZODB-5.6.0-py3.7.egg/ZODB/blob.py", line 689, in copyTransactionsFrom
    copyTransactionsFromTo(other, self)
  File "/Users/peter/workspace/project/buildout/eggs/ZODB-5.6.0-py3.7.egg/ZODB/blob.py", line 981, in copyTransactionsFromTo
    blobfilename = source.loadBlob(record.oid, record.tid)
AttributeError: 'RelStorage' object has no attribute 'loadBlob'

Without the blob-dirproperty I can export to filestorage without a problem but no blobs.

[jamadden]

RelStorage only has a loadBlob method if it is configured to support blobs. The AttributeError you're getting here means that the source relstorage configuration does not have blobs configured. You'll need to add that configuration.

[agitator]

So how should the zodbconvert config look like, if the blobs are stored directly in postgres (no shared blob) and I want to export that to a local filestorage with blob directory?

Or do I have to use the blob cache directory as source and will I get all blobs then?

[jamadden]

Yes, you need to activate blob support for your source. You need to configure it in exactly the same way you have it configured "normally", i.e., for your application. If you don't use a shared blob directory, but just a local cache, that's what you need to configure. Everything will be copied that way.

[agitator]

Thanx for clarification, will try it out!

[agitator]

Confirm it works! blod directory (layout zeochache) fills up with what looks like all blobs and zodbconvert creates the backup as expected.