Creating a common 3d model packaging format

[Floppy]

There doesn't seem to be a standardised way to distribute 3d models along with metadata, multiple files, etc. This seems like it would be a useful thing. The idea isn't to create yet another 3d format for actualy models, but to provide structured metadata and packaging to distribute these things.

A fairly easy off-the-shelf possibility for this is to use the datapackage format, which specifies a main file which includes metadata and then points to other resources, such as the models. It looks to me like it would work well for 3d files. We could define an extension like Tabular Data Package to add extra information or structure as required

For instance, here's an example datapackage.json file:

  "profile": "data-package",
  "resources": [
    {
      "name": "bunny",
      "path": "bunny.stl",
      "profile": "3d-model",
      "schema": {}
    }
  ],
  "keywords": [
    "stanford",
    "bunny",
    "datapackage"
  ],
  "name": "stanford-bunny",
  "title": "Stanford Bunny",
  "description": "The classic Stanford Bunny model, packaged up with Datapackage",
  "homepage": "https://graphics.stanford.edu/data/3Dscanrep/",
  "contributors": [
    {
      "title": "James Smith",
      "role": "packager",
      "web": "https://floppy.org.uk"
    }
  ],
  "licenses": [
    {
      "name": "CC-BY-SA-4.0",
      "title": "Creative Commons Attribution Share-Alike 4.0",
      "path": "https://creativecommons.org/licenses/by-sa/4.0/"
    }
  ]
}

There is a proposal around bundling datapackages into a single zipfile for distribution, which would work, though it's not in the standard yet. https://github.com/frictionlessdata/specs/issues/132

I could easily write a custom web app that helps make a valid JSON file that creators could use when they package up their models, or the generic one would also work fine: https://create.frictionlessdata.io/.

I think if we did do this, it would be a project alongside VanDAM, not part of this repo specifically, so this issue is just a place to collect thoughts, really.

[Floppy]

It's possible that 3MF can handle this, we just need better adoption of that spec. I'm having a read of https://github.com/3MFConsortium/spec_core/blob/master/3MF%20Core%20Specification.md#part-i-3mf-documents. Stream of consciousness follows, as I read:

https://github.com/3MFConsortium/spec_core/blob/master/3MF%20Core%20Specification.md#appendix-b2-3mf-metadata-example includes metadata, and the 3MF file can include multiple models, but I'm not quite sure yet how it hangs together.

If a 3MF file is intended inherently to be one "scene" with all objects in the same space, then it's not quite what we want to achieve with this, I don't think.

The model, in this specification, refers to the object or objects to be created via 3D manufacturing processes as a single operation. It might include a single object, multiple homogenous objects, multiple heterogeneous objects, an object fully enclosed in another object, or multiple objects in an interlocked and inseparable assembly.

That implies to me that each model, albeit composed of multiple objects, is intended to be printed all at once.

A payload that has a 3D Model root part is known as a 3D payload. There can be more than one 3D payload in a 3MF Document, but only one primary 3D payload.

The existence of a primary payload implies to me that a single file is intended to represent a single "thing". We want a way to collect together multiple things, so I think a single 3MF file doesn't quite fit what we want here.

[aneurinprice]

I think at least for the first version of this. We can safely ignore separate objects, e.g. Creator/Collections.

Ideally I'd suggest something similar to the way thingiverse handles their filestructure; Namely:

├── LICENSE.txt
├── README.txt
├── files
│   ├── Yuneec_Blank_Camera_Bracket_01.stl
│   ├── Yuneec_GoPro_v1.stl
│   └── Yuneec_Typhoon__Quarter-20_Camera_Bracket.stl
└── images
    ├── 776ddbca9f14ef2aa9781fff58cb73b6.png
    ├── 91a8076e2069232adef6810fac57e201.png
    ├── Yuneec_GoPro_Mount_01.jpg
    ├── Yuneec_GoPro_Mount_02.jpg
    ├── Yuneec_GoPro_Mount_03.jpg
    └── f4cefa80fc08767b729c215ab47fc11f.png

This format is about as simple as it gets, Our metadata could look something like:

Manyfold.meta

Name: 'Yuneec Typhoon H Universal Camera Mount!'
Creator: 'DIY3DTech'
Link: 'https://www.thingiverse.com/thing:2804742'
Licence: 'Creative Commons Attribution NonCommercial'
SummaryShort: 'The Yuneec Typhoon H is an amazing drone and with this wanted the ability to other camera payload then the stock CGO 3+ so I purchased a top plate'
SummaryFull: >
    This is a long paragraph
    that I am cutting down
    because it looks horrendous
    but you probably
    get my point
PreviewFile: 'images/Yuneec_GoPro_Mount_01.jpg'
Tree: #generated with `tree . -J`
  [
    {"type":"directory","name":".","contents":[
      {"type":"file","name":"LICENSE.txt"},
      {"type":"file","name":"Manyfold.meta"},
      {"type":"file","name":"README.txt"},
      {"type":"directory","name":"files","contents":[
        {"type":"file","name":"Yuneec_Blank_Camera_Bracket_01.stl"},
        {"type":"file","name":"Yuneec_GoPro_v1.stl"},
        {"type":"file","name":"Yuneec_Typhoon__Quarter-20_Camera_Bracket.stl"}
      ]},
      {"type":"directory","name":"images","contents":[
        {"type":"file","name":"776ddbca9f14ef2aa9781fff58cb73b6.png"},
        {"type":"file","name":"91a8076e2069232adef6810fac57e201.png"},
        {"type":"file","name":"Yuneec_GoPro_Mount_01.jpg"},
        {"type":"file","name":"Yuneec_GoPro_Mount_02.jpg"},
        {"type":"file","name":"Yuneec_GoPro_Mount_03.jpg"},
        {"type":"file","name":"f4cefa80fc08767b729c215ab47fc11f.png"}
      ]}
    ]}
  ,
    {"type":"report","directories":3,"files":12}
  ]

Reference: https://www.thingiverse.com/thing:2804742

For the SummaryFull field it might be nice to have this as markdown, but that is probably a problem for another day.

We could then write mappers for external sites to map their own data into our more generic structure.

Do you see anything immediately wrong with this?

[aneurinprice]

Ideally I'd like to keep any Manyfold spesific IDs out of this file. We can write a wrapper to map values like Creator to a manyfold ID, It would be nice to make this file as generic as possible.

Print settings could live in the Summary in the sake of genericness.

[Floppy]

Thanks for having a think through this! Some quick thoughts, I'll think more later though:

• I'd really like to use a common standard as a base if at all possible; the world doesn't need yet another metadata format. If JSON /datapackage feels too cumbersome to edit, we could use a YAML version which is close to what you've got there, and the two can be fairly easily converted between.
• We don't really need to specify a particular folder structure, the metadata file can contain whatever paths are necessary.
• We should use the SPDX standard notation for licenses (which Manyfold uses internally so should be easy)
• Agree keeping Manyfold IDs out of it, they're transient. Creator IDs will be a challenge there though, perhaps canonical URLs for them as an identifier or something might be the way forward.

[Floppy]

Looks like datapackage v2 came out a couple of months ago - I will have a look at the changes: https://datapackage.org/

[Floppy]

Other advantage of using the standard is that there are a bunch of existing parsers out there which will make it easier for others to adopt: https://datapackage.org/overview/software/

[aneurinprice]

the world doesn't need yet another metadata format.

/releveantxkcd

If JSON /datapackage feels too cumbersome to edit,

Thinking about it; This ideally will never be edited by hand, so readability isnt that much of a concern.

We don't really need to specify a particular folder structure, the metadata file can contain whatever paths are necessary

You are right. I think my bias comes from habit more than anything else, as images/ files/ is how I organise things.

Creator IDs will be a challenge there though, perhaps canonical URLs for them as an identifier or something might be the way forward.

I can think of a few ways we can do this, however none of them are nice. Currently torn between forcing a creator URL on every creator object in the app or just using the name of the creator and hoping for the best. Leaning towards the latter currently.

[mossholderm]

Looking at @aneurinprice 's example, it would also be helpful to have a mechanism to provide linkage between items. For example, you might want to say that a specific image is related to a specific STL file.

I assume you could achieve this in datapackage format using schema extensions and a little documentation. Something like this...

  "profile": "data-package",
  "resources": [
    {
      "name": "bunny",
      "path": "bunny.stl",
      "linked-images": [
        {
          "title": "Preview"
          "path": "./images/bunny-preview.jpg"
        },
        {
          "title": "Assembly Instructions"
          "path": "./images/bunny-assembly.jpg"
        }
      ],
      "profile": "3d-model",
      "schema": {}
    }
  ],
  "keywords": [
    "stanford",
    "bunny",
    "datapackage"
  ],
  "name": "stanford-bunny",
  "title": "Stanford Bunny",
  "description": "The classic Stanford Bunny model, packaged up with Datapackage",
  "homepage": "https://graphics.stanford.edu/data/3Dscanrep/",
  "contributors": [
    {
      "title": "James Smith",
      "role": "packager",
      "web": "https://floppy.org.uk"
    }
  ],
  "licenses": [
    {
      "name": "CC-BY-SA-4.0",
      "title": "Creative Commons Attribution Share-Alike 4.0",
      "path": "https://creativecommons.org/licenses/by-sa/4.0/"
    }
  ]
}

[fidoriel]

Would it be possible to also store this json in file system, so that the storage folder is useful without manyfold? For example for backup purposes and be more or less not dependent on the manyfold db?

Also to consider multiple file folders:

├── LICENSE.txt
├── README.txt
├── files
│   ├── With Logo
│   |   ├── Yuneec_Blank_Camera_Bracket_01_logo.stl
│   |   ├── Yuneec_GoPro_v1_logo.stl
│   ├── Without Logo
│   |   ├── Yuneec_Blank_Camera_Bracket_01.stl
│   |   ├── Yuneec_GoPro_v1.stl
└── images
    ├── 776ddbca9f14ef2aa9781fff58cb73b6.png
    ├── 91a8076e2069232adef6810fac57e201.png
    ├── Yuneec_GoPro_Mount_01.jpg
    ├── Yuneec_GoPro_Mount_02.jpg
    ├── Yuneec_GoPro_Mount_03.jpg
    └── f4cefa80fc08767b729c215ab47fc11f.png

I have got issues with models with many folders, they are recognizes as model name. I think with this issue, a lot of my current issues with manyfold could be solved.

[Floppy]

That's the idea, yes :)

[fidoriel]

Maybe image folder should be split into rendered preview images and real images. Having https://github.com/manyfold3d/manyfold/issues/1706 and possible thingiverse/printables imports in mind.