Is it possible to use persistent database instead of local storage for saving app settings?

[afanjul]

Hi, firstly thanks for your contribution, it's a great project 🙏

Is there anyway to use a centric persistent database to store the big-agi settings (like chats, api keys, folders, etc.) instead of using browser localstorage?

I see in the docs that I can integrate big-agi with a postgres but it's not clear if that is only for "chat sharing" or also to save app settings.

My Goal is to share the big-agi interface among some users, so we can see the same folders, models, settings, etc.

thanks

[afanjul]

To add more context to my question, I'm using big-AGI as a replacement for ChatGPT UI, so I installed big-AGI in a remote server, and it's working pretty fine! my problem is that once I move to another computer, or any other user enter the big-AGI interface, all the settings are "reset" because as I could see, the info is being stored in the browser local storage. So a easy solution would be just to store the same info in a common persisten centric place (a database connected, or even a json file in the server, etc.)

[afanjul]

I was thinking that for a quick solution by the moment, could you use storage.sync (instead of just storage.local) to sync the settings through computers using the same browser/chrome account? The interface for storage.sync is exactly the same as storage.local. Thanks

Update: I see that the chat conversations are not stored in localStorage any more (instead is using indexedDB), so the solution to use storage.sync is only partial, as I need to make persistent all the app data (settings, conversations, folders, etc.), but anyway it's a first step.

[enricoros]

could you use storage.sync (

Thanks for doing some research! This sounded promising indeed, but it's only available for Chrome Extensions, and it's not a widely supported web feature (we run on mobile, iOS, etc.) and again - just extension. But thanks for researching, that would have been a GREAT approach.

You're correct that right now Postgres (or Mongo) are only for the Chat Sharing feature. Big-AGI is right now a tool that saves data in the local machine (Chats are in the IndexedDB browser storage, everything else in the local storage). Transforming Big-AGI to a fully multi-user experience would take some work, which will make it more fit for enterprise use cases (and less for individual users use case). Haven't made the decision whether to move in the multi-user experience yet.

[pacoccino]

That's an interesting question. I love big-AGI but for me there something wrong in its design: storing data on client side and self hosting are paradoxal.

In the case of the hosted version on https://get.big-agi.com/ it totally makes sense: anyone can come, use the app with their own keys and keep everything private. But when self-hosting, we'd like to share settings/keys/convos across different device in our private network. What's the point of self hosting otherwise ?

[afanjul]

could you use storage.sync (

Thanks for doing some research! This sounded promising indeed, but it's only available for Chrome Extensions, and it's not a widely supported web feature (we run on mobile, iOS, etc.) and again - just extension. But thanks for researching, that would have been a GREAT approach.

You're correct that right now Postgres (or Mongo) are only for the Chat Sharing feature. Big-AGI is right now a tool that saves data in the local machine (Chats are in the IndexedDB browser storage, everything else in the local storage). Transforming Big-AGI to a fully multi-user experience would take some work, which will make it more fit for enterprise use cases (and less for individual users use case). Haven't made the decision whether to move in the multi-user experience yet.

could you use storage.sync (

Thanks for doing some research! This sounded promising indeed, but it's only available for Chrome Extensions, and it's not a widely supported web feature (we run on mobile, iOS, etc.) and again - just extension. But thanks for researching, that would have been a GREAT approach.

You're correct that right now Postgres (or Mongo) are only for the Chat Sharing feature. Big-AGI is right now a tool that saves data in the local machine (Chats are in the IndexedDB browser storage, everything else in the local storage). Transforming Big-AGI to a fully multi-user experience would take some work, which will make it more fit for enterprise use cases (and less for individual users use case). Haven't made the decision whether to move in the multi-user experience yet.

Thanks for your response @enricoros, and sorry for my wrong indications about storage.sync, I indeed used that feature in the past for some "personal extensions" I built, but surely I didn't take in account that is only for extensions... 😢

Maybe there is a easy solution, just using a localStorage "polyfill" but with a local file (probably a JSON file), something as simple as this:

const fs = require('fs');
const path = require('path');

const storageFilePath = path.join(__dirname, 'localStorage.json');

class LocalStoragePolyfill {
  constructor() {
    this.items = this.load();
  }

  load() {
    try {
      const data = fs.readFileSync(storageFilePath, 'utf8');
      return JSON.parse(data);
    } catch (error) {
      return {};
    }
  }

  save() {
    fs.writeFileSync(storageFilePath, JSON.stringify(this.items, null, 2), 'utf8');
  }

  getItem(key) {
    return this.items[key] || null;
  }

  setItem(key, value) {
    this.items[key] = value.toString();
    this.save();
  }

  removeItem(key) {
    delete this.items[key];
    this.save();
  }

  clear() {
    this.items = {};
    this.save();
  }

  get length() {
    return Object.keys(this.items).length;
  }

  key(index) {
    const keys = Object.keys(this.items);
    return keys[index] || null;
  }
}

// Exporting an instance to mimic the global localStorage object
module.exports = new LocalStoragePolyfill();

The same for "chat conversations".

This way will be easy to "share" the same app among multiple users.

[enricoros]

Got it. Different problem with this implementation. LocalStorage is a frontend (browser) technology, and you won't have access to backend (nodejs) objects such as 'fs' from there. I.e. browser can't write files that way.

[AIndoria]

Sidebar, but one method (on docker at least) would be to store everything on a local volume - at least for settings. Just storing everything on a local json should work that's mapped to whatever local path.

[monxas]

+1 to having a volume that stores data on the selfhosted version. I'm not sure if I'm using this correctly but Each time I want to use big-agi on a new computer I even have to add new api keys? is this expected? I think the selfhosted version would improve a lot with just a yaml or a json to have keys stored.

[enricoros]

@monxas this is a good point. Right now the keys are set on the server side as environment variables. So you feed the play your own big as yeah you can set those environment variables and no client will ever see the keys.

For a yaml solution, would you load that on the client side or server side? And do you have any examples that we can take a look at in modeling the solution?

[wdawson]

I'd also like a way to access chats from any browser/computer from my self-hosted solution.

Each time I want to use big-agi on a new computer I even have to add new api keys? is this expected? I think the selfhosted version would improve a lot with just a yaml or a json to have keys stored.

@monxas I do not have to do that with my set up (below). Note, I'm using a self-hosted solution with docker, not https://get.big-agi.com, where (at least from my understanding) you would need to add API keys for each browser you use.

I am self-hosting using docker-compose and have the following in my docker-compose.yaml (note the env_file):

    big-agi:
        container_name: ai-big-agi
        image: ghcr.io/enricoros/big-agi:latest
        env_file: ./big-agi/my.env
        ports:
          - 3000:3000

Then the env file is:

# Set OpenAI API Key
OPENAI_API_KEY="my-key-here"

Not sure if that will work for you or not, but works for me.

[joelmnz]

Adding my +1 for shared content. I want to self house and I change between about 3-4 devices regularly and want the chat data on all devices.

If we could set a dataSource api and key in yaml and in the browser have something like a guid for a "syncKey" to avoid having to setup the entire user management side of things, just have this for local hosting so you can share between your devices or have multi users locally with low chance of guessing the key.

This is an essential feature for me so keen to help with research or a POC

[joelmnz]

Just a thought, instead of a backing database, to keep this UI hostable and workable from the big-agi url what about how Draw.io works, allows users to nominate a cloud provider like OneDrive or GoogleDrive and uses their auth to allow users to save to a cloud folder.

Is that a simpler change than a back end database?

[ghost]

Also would really like to see this implemented, it is possible to use Google Drive Sync which is what BetterGPT does. Or at least an option for some local (on the server) storage, even if it's simple JSONL files or something. I don't really care about multi-user, but I use multiple devices and OSes myself, so not having shared chats is quite inconvenient.

Also https://chat-preview.lobehub.com/ (LobeChat) has experimental WebRTC sync support (which requires both devices to be online for the sync to happen), although that'd be a worse alternative for me.

[ovachiever]

Any thoughts towards using something like Supabase as a backend? That's what chatbot-ui did to move away from local storage and move to a true multiuser experience and it's real slick, seems easy enough to implement (talking out of my butt there, but).

[joelmnz]

I have started on a "big-agi-syncserver" which will expose API endpoints to call and can store back end data where ever (currently I'm just adding SQLite and SQL Server but extensible for any back end storage type).

Changes I would like to make in this big-agi project are

• Add 2 user settings 1. "Sync Server Url", 2. "Sync Key" (possible in the Preferences > Tools area as "Sync Server" e.g. like the Browsing & Google Search API settings)
• include a "Test" button next to these settings which will call make a test GET http request to "{url}/getLastSync/{syncKey}" showing success or error
• Add a "Sync with Server" button to the "Export" popup that will sync the data with the server (to be define).

Is anyone able to validate if this is something this project wants and if creating a separate big-agi-syncserver project is suitable. Its essentially an integration & would be controlled by setting/variable so when self hosting you can turn this on/off. Sync-Server would be user hosted.

[joelmnz]

Any thoughts towards using something like Supabase as a backend? That's what chatbot-ui did to move away from local storage and move to a true multiuser experience and it's real slick, seems easy enough to implement (talking out of my butt there, but).

Had a look at Superbase, I haven't used it before but it looks simple enough and the free plans should be fine for single private use. Will have a play and post back here. Also, the setup would be close to how I have the big-agi-syncserver working so might be able to do both.

[joelmnz]

@enricoros can you comment on your preferred approach here?

Is it best to use the existing Postgre DB with PrismaDB or keep this feature separate and store conversation data in a user selected location like custom api or Supabase?

[joelmnz]

UPDATE: I have an initial version working to sync Conversations to Supabase which can be shared across devices (no users, 1 Supabase = 1 set of chat history).

Here is the PR into my forked repo https://github.com/joelmnz/big-AGI/pull/1

Changes

Added User based settings for Sync details

Added a manual "Sync" button to the "Export" screen

Low error handling, requires Supabase project database setup with the expected schema (see README.md in the src/modules/supabasesync/README.md folder for instructions.

[joelmnz]

PR Ready, its not that nice a solution, I'd be grateful for someone to suggest a better way to sync conversations

https://github.com/enricoros/big-AGI/pull/554

[shuther]

to add my 2 cents, maybe remote storage is easier to implement and easier to use (above all if we can have the choice to keep it local or remote? Supabase has a lot of requirements to the project.

[joelmnz]

Thanks @shuther that project looks great, It has the flexibility of back end storage I was looking for, will give that a go, it might fit better for user settings etc and means we can keep same domain models.

@enricoros , I think this might be a better option that Supabase Sync, thoughts?