alpstable / gidari

Transport web data to local/remote storage using Gidari
Apache License 2.0
29 stars 29 forks source link

Make the connectionStrings field an array of maps #180

Closed prestonvasquez closed 1 year ago

prestonvasquez commented 2 years ago

Right now the connectionStrings type for a configuration YAML file is an array of strings. We may need to add more information / context to each value in this array so it should be converted into an array of maps:

url: someURL
storage: 
  - connectionString: x
  - connectionString: y

This should be fairly straight-forward to do. It would require

  1. Updating these yml files with the new format: https://github.com/alpine-hodler/gidari/tree/main/internal/transport/testdata/upsert

  2. Add a new struct to transport.go called storageReference (or whatever):

type storageReference struct {
  ConnectionString: string `yaml:"connectionString"`
}
  1. Updating the Config struct to use []storageReference rather than []string for Conifg.ConnectionStrings. We should also change the name from Config.ConnectionStrings to Config.StorageReference.
axlts commented 2 years ago

Hello, I would like to do this! @prestonvasquez can you assign me?

krishnaindani commented 2 years ago

Could you please assgn to me as we talked about combining with other issue?

prestonvasquez commented 2 years ago

@axlts Thank you for your interest in this issue, however it is directly related to work @krishnaindani has in progress. These issues are fairly similar, dealing with the same area of code: https://github.com/alpine-hodler/gidari/issues/153 , https://github.com/alpine-hodler/gidari/issues/155 .

krishnaindani commented 2 years ago

would we like to take like

storage:
  - connectionString: dbtype

for example:

storage:
  - "mongodb://mongo1:27017/jokes": mongoDB
  - "mongodb://mongo2:27017/jokes": mongoDB
krishnaindani commented 2 years ago

Sorry I think I got it wrong, the intention here is to keep the nested name to still connectionString just the value needs to be the url of the connection, is it? Just bit of confused here

for example

url: someURL
storage: 
  - connectionString: "mongodb://mongo1:27017/jokes"
  - connectionString: "mongodb://mongo2:27017/jokes"
prestonvasquez commented 2 years ago

@krishnaindani The storage package already has a method to parse the scheme from connection string: https://github.com/alpine-hodler/gidari/blob/main/internal/storage/storage.go#L90

Using an example from Postman's public API list, I think it should look something like

url: https://official-joke-api.appspot.com
storage: 
  - connectionString: mongodb://mongo1:27017/jokes
  - connectionString: mongodb://mongo1:27017/jokes
rateLimit:
  burst: 5
  period: 1
requests:
  - endpoint: /random_joke
prestonvasquez commented 2 years ago

@krishnaindani Are you still interested in pursuing this issue?

krishnaindani commented 2 years ago

sorry I was in gophercon, so I missed it. Someone can talk a look before I can catch up maybe.

prestonvasquez commented 1 year ago

Closed by #360