Next gen architecture

[ctengel]

POST /download (i.e. allow fully assisted downloading) - need "worker bees" close to obj store

• URL
• (opt) URL_referer
• (opt) metadata
• (opt) SW
• (opt) myhost

POST /upload

• i.e. pass thru to obj store, let webgui upload, etc

GET /file

• actually return instead of redir

flexibility

• don't assume bucket name is consistent across the board
• store buckets and what is where in db
• allow multiple servers
• allow passthru (download for me, upload to me, download via me)
• permissions (i.e. no need for seperate instances)

Database

object

• System (definetely removed at this point)
• Bucket (link to bucket table)
• Path
• UUID
• Size
• Hash
• Date
• ObjMeta
• ObjTags
• statuses
  • discovered T/F
  • informed T/F
  • facilitated upload T/F
  • completed T/F
  • preinformed T/F
  • deleted T/F
  • self DL T/F

file

• ObjUUID
• instUUID
• URL
• URL_referer
• Metadata JSON
• UL user (becomes a link to user table)
• UL_sw
• UL_host
• mtime (orig source)
• date-indexed

user is a table

user_buck (permissions)

• can_read
• can_write
• user
• bucket

bucket

• name

buck_serv

• bucket
• server
• writable
• read_preferable
• name_on_server

server

• host
• ip
• status
• creds

[ctengel]

Need more RPis

4 pricing

• 2GB $35
• 4GB $55
• 8GB $75
• "400" $70 modified 4GB with keyboard but missing some others

Note the Zero also.. $15???

DigiKey

[ctengel]

Some authentication into the system

https://boto3.amazonaws.com/v1/documentation/api/1.9.42/reference/services/s3.html#S3.Client.generate_presigned_url

[ctengel]

CM4???

• https://datasheets.raspberrypi.com/cm4io/cm4io-product-brief.pdf
• https://datasheets.raspberrypi.com/cm4/cm4-product-brief.pdf
• https://datasheets.raspberrypi.com/cm4io/cm4io-datasheet.pdf
• https://datasheets.raspberrypi.com/cm4/cm4-datasheet.pdf

cluster

• https://will127534.github.io/Miniature-CM4-Cluster/
• https://deskpi.com/products/deskpi-super6c-raspberry-pi-cm4-cluster-mini-itx-board-6-rpi-cm4-supported
• https://www.kickstarter.com/projects/ultimate-rpi-cluster/rpi-cm4-cluster-prototype
• https://turingpi.com/
• https://uplab.pro/2022/03/compute-blade-changelog/
• https://uplab.pro/2021/10/mincab/

other clusters https://uplab.pro/2020/12/raspberry-pi-server-mark-iii/

[ctengel]

https://github.com/hoppscotch/hoppscotch

[ctengel]

consider multinode, multidrive https://min.io/docs/minio/linux/operations/install-deploy-manage/deploy-minio-multi-node-multi-drive.html

[ctengel]

consider securing with HTTPS https://www.linkedin.com/pulse/build-your-own-private-cloud-home-raspberry-pi-minio-huerta-arias

[ctengel]

Maybe we need to do this with #22 to complete #13 / #27 ?

[ctengel]

Do we need to assume S3 bucket is the only storage mechanism?

Do we want to track each object at exactly one place/bucket?

IE do we want to copy an object in some circumstances? Or do we only care about it in one place, the most readily accessible place?

[ctengel]

I think the big thing really is does object management happen at the object layer ever or just bucket? If the entire bucket is replicated or backed up at once then it is simple... But if one object gets archived from the bucket then what?

[ctengel]

leverage pqsql schemas