Open graykode opened 4 years ago
How about this ER Diagram?
@seongpyoHong
There are opinions about several DB types.
bucket id
is hashed here and used as a string type. Therefore, it should be expressed as a string rather than an integer.filetype
is a list, and even if it is converted to a string, it exceeds at least 255 characters. Therefore, another type of alternative is needed.additional
is a dict type, but like filetype, it exceeds at least 255 characters even when converted to string. Therefore, a type such as Text seems more appropriate.Additionally, for the attribute name, It's better to use pothole notation(lowercase letters and underbar).
@graykode
varchar
to text
and naming convention to pothole notation.@seongpyoHong I've fixed ER Diagram like below:
CREATE TABLE bucket (
id varchar(255) primary key not null,
additional text not null,
dataset_name varchar(255) not null,
endpoint varchar(255) not null,
compressor varchar(255) not null,
sagemaker boolean not null default false
);
CREATE TABLE files(
id serial primary key not null,
name varchar(255) not null,
bucket_id varchar(255),
constraint bucket_id foreign key (bucket_id) references bucket(id)
);
CREATE TABLE attributes (
id serial primary key not null,
name varchar(255) not null,
type varchar(255) not null,
shape varchar(255) not null,
itemsize integer not null,
bucket_id varchar(255),
constraint bucket_id foreign key (bucket_id) references bucket(id)
);
CREATE TABLE indexer (
id serial primary key not null,
indexer_end bigint not null,
length integer not null,
name varchar(255) not null,
bucket_id varchar(255),
constraint bucket_id foreign key (bucket_id) references bucket(id)
);
Since filetype
is a list
type that is frequently modified, so I decided that it would be better to make this attribute into one table.
As one minor addition, I know that variable-length strings (Text) can slow down the DB. So, how about setting an additional
attribute that only uses Text type to very large n, varchar(n)?
Versions prior to 0.4.0 manage metadata in json format, so using only s3 could maintain the shape. However, since metadata is managed by the RDBMS, it is necessary to write RDS code for AWS RDMS.
Now metadata attribute is managed as a JSON file. However, as a long-term plan, we will modify it to work concurrency with the database.
e.q : data