search

bevyengine / bevy

A refreshingly simple data-driven game engine built in Rust
https://bevyengine.org
Apache License 2.0
36.15k stars 3.57k forks source link

Better Asset System: Externals Part 1 - Locator #2710

Open VVishion opened 3 years ago

VVishion commented 3 years ago

Introduction

The Engine needs to interoperate with other systems, such as:

It does so by getting input from and giving output to those systems in constructs provided by these systems, such as:

They are often referred to as Resources, but …

We may call them Externals.

What is an External?

The Engine interacts with Externals for the means of input and output.

How is an External interacted with?

Every system works differently, has different kind of constructs and therefore different interface(s). We may call an abstraction over an interface of a system Protocol.

Where do Externals exist?

Externals exist in their system. They may be hierarchically structured within their system. Relevant Externals are mostly local - grouped together, such as in the application directory in the file system, or a superior endpoint of an http(s) server. Related or alike Externals may also be local, such as Assets, Scripts, …; Models, Scenes, …

Requirements

The representation must allow definition of:

It must also allow trivial transformations to established representations, such as file paths, URLs, …

Proposal

This is the first of hopefully four indepently applicable proposals. They aim to split Externals and Assets, while trying to improve the Terminology, Structure and/or API for and between them. They should be applicable for both Engine Runtime and Suite.
Some terminology and functionality is borrowed from the URL standard.

How is an External located?

An External is located by a Locator and its components scheme and path.

What is a Locator?

A Locator describes a Location and how to access it.

Locator Components

Scheme

Identifies the Protocol.


Path

Location of the External. Every Locator detail other than scheme and fragment belongs to path.

Representations of path are scheme-specific, except the character / has reserved meaning.


For the file scheme this would be just: path

For the http(s) scheme this would be: domain/ip:port + path

…

Further distinction within path must be done by other in-Engine libraries if necessary.


How is part of an External located?

A part of an External is located by the Location of an External and a fragment.

Fragment

Locates part of an External. We may call parts of an External Fragments from now on.

Not every External has Fragments.

If no fragment was defined the Locator locates an External, a Fragment otherwise.

Note: A file is not a Fragment of a directory. Both file and directory are Externals. Distinction may not be clear.


What is an Asset?

An Asset is a deserialized External or Fragment.

Representation

A Locator is represented by

scheme :// path # fragment

Examples

file:///path/to/app/path/to/file.ext#fragment
└─┬┘    └────────────┬─────────────┘ └──┬───┘
  │                  │                  │
scheme              path             fragment
http://192.178.1:8080/path/to/file.ext#fragment
└─┬┘   └──────────────┬──────────────┘ └──┬───┘
  │                   │                   │
scheme               path              fragment
         path
          │
          │
  asset:///#assetid
  └─┬─┘     └──┬──┘
    │          │
scheme      fragment

Relative Locations

A Locator can represent a relative Location. This is scheme-specific.

A Locator does not know whether it is absolute or relative. This is scheme-specific. Wrongfully Joining an absolute Locator to a relative Locator without fragment cannot be caught as an error by Locator.

Locations can only be split into multiple Locators at instances of :// or /, scheme must always be suffixed by :// and fragment always be prefixed by #.

Locators defining a scheme can be relative to another Locator with the same scheme - not only absolute Locators can define a scheme.

Examples

file:///path/to/app
file://path/to/file.ext
http://path/to/file.ext
file://
path
file.ext
#fragment

Joining Relative Locators

Relative Locators must have either the same scheme or one or both have none. Locators with a fragment each cannot be joined. Minor Normalization might be applied after Joining.


Normalization

Locator may feature minor Normalization by replacing contiguous sequences of /s in path with just one /. If / is the last character of path, truncate it by one character.

Further Normalization is scheme-specific.

To limit representation variaties of Locator and therefore handling cases for consumers Normalization is desired, but may also be delegated completely.


URLs Problem

For unknowing developers representations such as:

file://path/to/file.ext, or path/to/file.ext

might describe a reasonable Location. But adhering to the URL specification means that "path" is the host and "/to/file.ext" is the path.


Correct would be:

file:///path/to/file.ext, and /path/to/file.ext


So whats different? path must be prefixed with / to indicate the end of host.

Most implementations know the file scheme specifies no host and will treat both representations equally. For the http(s) scheme both representations cannot be treated equally.

So why is this not a problem within Locator?

Locator delegates distinctions within path. A relative Locator's path must therefore not be prefixed with /, but can be. Absolute paths must still adhere to their specifications and schemes where there is an expected part before the first / for it's path like http(s) are not freed from this problem. But relative paths can freely start with or without /.

Therefore, joining a Locator with relative Locators from either path/to/file.ext or /path/to/file.ext result in equal joined Locators by minor Normalization or consumers are responsible to treat one / or a contiguous sequence of /s equally to uphold this guarantee.

Since game developers will mostly always use relative Locators this problem is likely hidden from them.


Location Local

In-Engine systems, such as the External or Asset system, may define a Location Local. All relative Locations are then assumed to be relative to the Location Local within the system.


Instructor

Locator may feature URLs Query syntax ? or something similar to support for example custom asset pipelines. Since its applications are generic, usage depends on consumers.

I tend towards waiting for other asset related systems to be prototyped or decided upon before deciding for a standard here. E.g.:

VVishion commented 3 years ago

Extends #2622

LegNeato commented 3 years ago

Locator reminds me of https://github.com/multiformats/multiaddr, used in libp2p: https://docs.rs/libp2p/0.39.1/libp2p/index.html

VVishion commented 3 years ago

They seem to be a bit similar, but have other requirements. I haven't taken inspiration other than the URL specification, though, since it is the most well-known and bevy already showed syntactical usage of it, such as the # separator.