[![Contributors][contributors-shield]][contributors-url] [![Forks][forks-shield]][forks-url] [![Stargazers][stars-shield]][stars-url] [![Issues][issues-shield]][issues-url] [![Issues][issuesclosed-shield]][issuesclosed-url] [![Kanban Board][board-shield]][board-url] [![MIT License][license-shield]][license-url] [![Code Coverage][code-shield]][code-url]

Synthetic File System

The student team is to develop a new file system that provides a unified view ona pre-configured set of existing file systems. The new virtual file system is to:
●Be implemented as a Linux FUSE (filesystem in user space) module

●Be POSIX compliant; initially, only read-only functions are needed

●Integrate NTFS, ext4, and S3 object stores

●Be able to filter which files are shown using meta-data

●Be configured from a configuration file during startup time





Test cases are:

●Browsing using an existing Linux file system browser (e.g. Nautilus)

●Using cp to copy from the new virtual file system



amos-ss2021-synthetic-file-system Bug · Requested Features

Table of Contents

1. About The Project
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. Roadmap
5. License
6. Contact

## About The Project This project implements a so called [FUSE](https://en.wikipedia.org/wiki/Filesystem_in_Userspace), a filesystem in userspace, in order to give a unified filesystem view on data in a [Metadatahub](www.metadatahub.de) database. The FUSE also allows for filtering the files it will show according to their metadata ### Product Vision The rapidly growing mass of data requires further refinements and new technologies in order to find the right data in this deluge of information. The creation and management of metadata is decisive for representing the content of stored objects and files. This metadata is held in a database for instant retrieval. Lists can be constructed from these databases to find files and objects in general, but they do not yet afford access to the real data. For this reason, a synthetic file system is considerably useful. A synthetic file system enables to access data, chosen by queries in the database of metadata. Retrieval of project-related data is done through a distributed database and a virtual file system that permits a single namespace for all relevant data designated by their metadata. ### Project Mission Due to the current Corona pandemic, as much data as possible is to be analyzed and evaluated with the help of artificial intelligence (AI). This requires a central intelligence to collect and interpret all accessible data distributed over several facilities. The key issue is that the data is organized and saved in different systems according to different storage types, structures, formats and criteria. The task, or rather mission is now to first make the data obtainable or readable via a uniform mechanism within the project time-frame. This would have the benefit of not having to adapt each application to the different memory types. The synthetic file system is a so-called adapter for each type of memory, so that a unified namespace can be formed from it. --- ## **Getting Started** ### Prerequisites * git * docker: [follow instructions for your platform](https://docs.docker.com/get-docker/) (Only needed if you want to try the fuse in the docker) * docker-compose (Only needed if you want to try the fuse in the docker) * [Metadatahub](www.metadatahub.de) ### Installation 1. Clone the amos-ss2021-synthetic-file-system ```sh git clone https://github.com/amosproj/amos-ss2021-synthetic-file-system.git ``` 2. If you want to try the FUSE in a docker environment, build it with ```sh cd amos-ss2021-synthetic-file-system docker-compose up --build ``` For subsequent uses you can just run ```sh docker-compose up ``` Then you can connect to the docker from a different shell via ```sh docker exec -it synthetic-file-system tmux ``` where ```tmux``` can also be replaced by ```zsh``` or ```bash```, depending on your preferences. 3. This fileystem will be mounted under ```~/fuse_mount```. Create this folder for later use: ```sh mkdir ~/fuse_mount ``` --- ## Usage ### Basics 1. SFS pulls its information from a running [Metadatahub](www.metadatahub.de) service. For more information or documentation refer to the vendor. 2. To run the SFS without docker, simply run ```sh python3 -m src mount_point ``` where ```mount_point``` is an empty folder you specify by yourself. 3. Warning: SFS with docker is bugged in the current release version and will be resolved soon 4. Attention when start the FUSE using docker: Since the FUSE blocks the current terminal, a new terminal in the docker has to be opened. For this you can just open a new terminal on the host and connect it again to the docker via ```docker exec -it synthetic-file-system tmux ```, or use tmux in the docker to open a new terminal (```ctrl+b -> ctrl+%```). For more information please refer to the [tmux documentation](https://github.com/tmux/tmux/wiki). 5. Traverse the virtual filesystem via a terminal, or via any file browser like ```nautilus```. For docker: The docker container is configured to support X-forwarding, so any UI program opened on the docker will be forwarded to the host. So to traverse the filesystem using a file browser under the docker, just run ```sh nautilus ``` from a terminal that is connected to the docker. ### Configuration SFS enables the filtering of files via their metadata. For this, a config file ```config/config.graphql``` is used. When starting SFS, all the filters will be read from this file and applied. More information regarding graphl syntax can be found [here](https://metadatahub.de/documents/graphql/#/definitions/ResultSet). **NOTE** The way the dynamic configuration works, it is needed, that the config file is written to and closed properly, just for example pressing ```ctrl+s``` won't always work. So when ```VIM``` for example, close the file with ```:wq``` or ```:x```. ----------------------------------------------------------------------- ## Roadmap See the [open issues](https://github.com/amosproj/amos-ss2021-synthetic-file-system/issues) for a list of proposed features (and known issues). ## License Distributed under the MIT License. See [LICENSE](https://github.com/amosproj/amos-ss2021-synthetic-file-system/blob/main/LICENSE) for more information. ## Contact AMOS PROJECT - amos-fau-proj3@group.riehle.org Industry Partner - [GRAU Data](https://www.graudata.com/) Project Link: [https://github.com/amosproj/amos-ss2021-synthetic-file-system](https://github.com/amosproj/amos-ss2021-synthetic-file-system) [contributors-shield]: https://img.shields.io/github/contributors/amosproj/amos-ss2021-synthetic-file-system.svg [contributors-url]: https://github.com/amosproj/amos-ss2021-synthetic-file-system/graphs/contributors [forks-shield]: https://img.shields.io/github/forks/amosproj/amos-ss2021-synthetic-file-system.svg [forks-url]: https://github.com/amosproj/amos-ss2021-synthetic-file-system/network/members [stars-shield]: https://img.shields.io/github/stars/amosproj/amos-ss2021-synthetic-file-system.svg [stars-url]: https://github.com/amosproj/amos-ss2021-synthetic-file-system/stargazers [issues-shield]: https://img.shields.io/github/issues/amosproj/amos-ss2021-synthetic-file-system.svg [issues-url]: https://github.com/amosproj/amos-ss2021-synthetic-file-system/issues [issuesclosed-shield]: https://img.shields.io/github/issues-closed/amosproj/amos-ss2021-synthetic-file-system.svg [issuesclosed-url]: https://github.com/amosproj/amos-ss2021-synthetic-file-system/issues?q=is%3Aissue+is%3Aclosed [board-shield]: https://img.shields.io/badge/Kanban-Board-grey?logo=data:image/svg%2bxml;base64,PHN2ZyB2aWV3Qm94PSIwIDAgMjQgMjQiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+PHBhdGggZD0iTTEzLjM1MiAxNC41ODVsLTQuNTA5IDQuNjE0LjcyLTQuMDYyTDMuNDI4IDcuNTcgMCA3Ljc1MyA3LjU4IDB2Mi45NTNsNy4yMTQgNi42NDYgNC41MTMtMS4xMDUtNC42ODkgNC45ODJMMjQgMjRsLTEwLjY0OC05LjQxNXoiLz48L3N2Zz4= [board-url]: https://github.com/amosproj/amos-ss2021-synthetic-file-system/projects/1 [license-shield]: https://img.shields.io/github/license/amosproj/amos-ss2021-synthetic-file-system.svg [license-url]: https://github.com/amosproj/amos-ss2021-synthetic-file-system/blob/main/LICENSE [code-shield]: https://codecov.io/gh/anuraghazra/github-readme-stats/branch/master/graph/badge.svg [code-url]: https://github.com/amosproj/amos-ss2021-synthetic-file-system/github-readme-stats