search

owncloud / data_exporter

Export/Import for ownCloud user data
GNU General Public License v2.0
7 stars 5 forks source link

Data Format V2 #77

Closed micbar closed 5 years ago

micbar commented 5 years ago

Extend the current Data Format to meet new requirements

Requirements

Directory Structure

/tmp/export/
├── User1
│   ├── files
│   │   └── files
│   │       ├── Folder
│   │       │   ├── Subfolder
│   │       │   │   ├── Screenshot_2019-03-28\ Settings\ -\ ownCloud.png
│   │       │   │   └── config_report_20190326.json
│   │       │   └── version-log.txt
│   │       └── welcome.txt
│   └── metadata.json
├── User2
│   ├── files
│   │   └── files
│   │       ├── Folder
│   │       │   ├── Subfolder
│   │       │   │   ├── Screenshot_2019-03-28\ Settings\ -\ ownCloud.png
│   │       │   │   └── config_report_20190326.json
│   │       │   └── version-log.txt
│   │       └── welcome.txt
│   └── metadata.json
├── admin
│   ├── files
│   │   └── files
│   │       └── welcome.txt
│   └── metadata.json
└── instancedata.json

Metadata

{
  "date": "2019-04-16T14:55:28+00:00",
  "originServer": "http://localhost/",
  "exporterFormatVersion":"2",
  "instanceId" : "ocmio7vr8e9g",
  "passwordsalt" : "ISabX7n8nHsgiYtVnPZ/umYSuu5PPW",
  "secret" : "dMaKhD75v1Y22TSfO8ekBAv41vvh/IZYGpyw+v8mJOqTn48w",
  "config" : [
    {
      "someConfig" : "Value"
    }
  ]
  "groups" : [ 
      "people",
      "someGID"
  ]
}
micbar commented 5 years ago

cc @IljaN

butonic commented 5 years ago

After discussing this with @micbar I have a few notes:

General layout

/tmp/export/
├── instance
│   ├── groups.jsonl
│   ├── instance.json
│   └── roles.jsonl
└── users
    ├── admin
    │   ├── files
    │   │  └── welcome.txt
    │   ├── comments.jsonl
    │   ├── files.jsonl
    │   ├── properties.jsonl
    │   ├── shares.jsonl
    │   ├── tags.jsonl
    │   └── user.json
    ├── User1
    │   ├── comments.jsonl
    │   ├── files.jsonl
    │   ├── properties.jsonl
    │   ├── shares.jsonl
    │   ├── tags.jsonl
    │   └── user.json
    └── User2
        ├── files
        │   ├── Folder
        │   │   ├── Subfolder
        │   │   │   ├── Screenshot_2019-03-28\ Settings\ -\ ownCloud.png
        │   │   │   └── config_report_20190326.json
        │   │   └── version-log.txt
        │   └── welcome.txt
        ├── comments.jsonl
        ├── files.jsonl
        ├── properties.jsonl
        ├── shares.jsonl
        ├── tags.jsonl
        └── user.json

instance data

Some data is needed only once:

user specific data

Every user gets his own folder with:

im - / export

federated shares

notes

IljaN commented 5 years ago

Regarding permissions @pmaier1 correctly objected that we shouldn't store them as "owner","viewer","editor",... because these are abstractions consisting of multiple primitive permissions i.e (edit,view,delete...). There is also the type "custom" which can consist of any permissions combination. So the only approach here is to work with the lower-level primitive permission.

Need also to create a matrix how the new perms map to the old ones.

butonic commented 5 years ago

closing as the new format PR has been merged. The current export looks like this:

.
└── einstein
    ├── files
    │   └── files
    │       ├── Documents
    │       │   └── Example.odt
    │       ├── Photos
    │       │   ├── Paris.jpg
    │       │   ├── San\ Francisco.jpg
    │       │   └── Squirrel.jpg
    │       └── ownCloud\ Manual.pdf
    ├── files.jsonl
    ├── shares.jsonl
    └── user.json
cat einstein/files.jsonl
{"type":"folder","path":"\/files","eTag":"5d9c925ed57b6","permissions":31,"mtime":1570542174}
{"type":"folder","path":"\/files\/Documents","eTag":"5d9c8e41a5ee2","permissions":31,"mtime":1570541121}
{"type":"file","path":"\/files\/Documents\/Example.odt","eTag":"8c883978b40a0119cded0d98f9f9f7f4","permissions":27,"mtime":1570541121}
{"type":"folder","path":"\/files\/Photos","eTag":"5d9c8e41efd41","permissions":31,"mtime":1570541121}
{"type":"file","path":"\/files\/Photos\/Paris.jpg","eTag":"b4071f830567be27c6806e3fe6afc2ab","permissions":27,"mtime":1570541121}
{"type":"file","path":"\/files\/Photos\/San Francisco.jpg","eTag":"2c3abec14b68a71899b21168ef86ec7b","permissions":27,"mtime":1570541121}
{"type":"file","path":"\/files\/Photos\/Squirrel.jpg","eTag":"fec2104bfc64bf7366822902fb66a2be","permissions":27,"mtime":1570541122}
...
{"type":"folder","path":"\/files\/core-10.3.0RC1","eTag":"5d9c925ed57b6","permissions":31,"mtime":1570542174}
...
{"type":"file","path":"\/files\/ownCloud Manual.pdf","eTag":"1038edfe0158e3899ed5079952210bcb","permissions":27,"mtime":1570541122}
cat einstein/shares.jsonl | jq
{
  "path": "/core-10.3.0RC1",
  "shareType": "user",
  "owner": "einstein",
  "sharedBy": "einstein",
  "sharedWith": "marie",
  "permissions": 31,
  "expirationDate": null,
  "password": null,
  "name": null,
  "token": null
}
cat einstein/user.json | jq
{
  "date": "2019-10-08T19:34:55+00:00",
  "originServer": "http://localhost/",
  "user": {
    "userId": "einstein",
    "displayName": "Albert Einstein",
    "email": "einstein@example.org",
    "passwordHash": "1|$2y$10$SpvL5K/1.zevs/sleDYk/udThqy2/o9WTL1N8zFLcfGrLpujmqtw2",
    "quota": "default",
    "backend": "Database",
    "enabled": true,
    "groups": [
      "physics-lovers",
      "sailing-lovers",
      "violin-haters"
    ],
    "preferences": [
      {
        "appId": "core",
        "configKey": "timezone",
        "configValue": "Europe/Berlin"
      },
      {
        "appId": "firstrunwizard",
        "configKey": "show",
        "configValue": "0"
      }
    ]
  }
}

Only thing I noticed: get rid of the /files prefix for all files in files.jsonl. That way the path in shares.jsonl matches the one found in files.jsonl. Tracked in https://github.com/owncloud/data_exporter/issues/111

:rocket:

create new issues for data format issues.