Open Stvad opened 1 year ago
So let's say promnesia module takes in a JSON-like config object (instead of specific config fields, it can't interpret them anyway):
SOURCES = [
Source(roamresearch, hpi_config={"export_path": "path/to/json", "error_policy": "throw"}),
]
Then the promnesia module could do some dynamic magic ✨ inside and set the corresponding HPI config, roughly something like import my.config; my.config.roamresearch = convert_json_to_object(hpi_config)
.
There are however some issues with this approach. It might work well in simple cases like roamresearch
. Its hpi module is my.roamreasearch
, its hpi config.py section is class roamresearch: ...
, so it's straightforward.
Consider something more complicated like twitter module:
https://github.com/karlicoss/promnesia/blob/master/src/promnesia/sources/twitter.py#L9-L11
In promnesia it's just importing my.twitter.all
and using whatever it spits out -- pretty simple.
However if you wanted to make promnesia.sources.twitter
somehow configure HPI for itself, this would be quite tricky.
It's hard to say what HPI modules correspond to it!
Not only in the sense that it's not very well documented -- but it might actually depend on 'subsources' configuration and user's HPI overlays.
E.g. the default HPI is this: https://github.com/karlicoss/promnesia/blob/master/src/promnesia/sources/twitter.py#L9-L11 -- it's trying to merge twitter GDPR archive + Twint export. These would correspond to twitter_archive
and twint
sections in the config.
However, in my personal overlay I'm also using data from Talon app on top of that https://github.com/karlicoss/hpi-personal-overlay/blob/master/src/my/twitter/all.py#L10-L13
... and the config section for Talon is twitter.talon
.
So that means hpi_config would have to take multiple config sections, something like this:
SOURCES = [
Source(twitter, hpi_configs=[
{"twitter_archive": {"export_path": ...}},
{"talon" : ...},
{"twitter.twint" : ...},.
]),
]
To be honest it looks less bad than I expected before I started typing the response, so maybe it's a feasible option after all 😅
However perhaps more realistic is making it something like this:
class hpi_config:
class roamresearch:
export_path = "/path/to/json"
class twitter_archive:
export_path = ...
class twitter:
class twint:
export_path = ....
....
<some magic helper to load hpi_config object into actual hpi>, e.g. hpi.load_extra_config(hpi_config)
SOURCES = [
Source(roamresearch),
Source(twitter),
]
I guess this is still simple enough, presumably the most annoying bit is having to init a separate config file, but if it's in the same file it's better? This way it would keep promnesia modules intact, and agnostic of exact HPI module internals.
I actually wonder if it's possible to write this magic config loader even within the current HPI/promnesia implementation (although need to experiment to find out). There are a couple of examples of generating configs dynamically in runtime
https://github.com/karlicoss/HPI/blob/4a04c09f314e10a4db8f35bf1ecc10e4d0203223/tests/config.py https://github.com/karlicoss/HPI/blob/5ef277526577daaa115223e79a07a064ffa9bc85/tests/demo.py https://github.com/karlicoss/HPI/blob/8422c6e420f5e274bd1da91710663be6429c666c/my/core/cfg.py#L31
So maybe something like this:
class hpi_config:
.... # same as above
import my.config
for key in dir(hpi_config):
# this likely won't work, might need something more elaborate)
setattr(my.config, key, getattr(hpi_config, key))
SOURCES = [
S(roamresearch),
S(twitter),
]
Apologies for a delayed response!
I originally was thinking about something closer to the latter approach where you can define the HPI configs in a way that you usually define them and Promnesia can understand and consume that (presumably including installing the HPI dependency if it's not doing that already)
Note to self -- been playing with it today, and at the moment it's achievable with something like this (needs improvement of course, but at least it's already possible in principle)
def reset_modules() -> None:
'''
A hack to 'unload' HPI modules, otherwise some modules might cache the config
'''
import sys
import re
to_unload = [m for m in sys.modules if re.match(r'my[.]?', m)]
for m in to_unload:
del sys.modules[m]
def brindex(path: PathIsh):
reset_modules()
from my.core.cfg import tmp_config
with tmp_config() as cfg:
from my.core.common import classproperty, Paths, get_files
class browser: # adhoc user-defined config
class export:
@classproperty
def export_path(cls) -> Paths:
return get_files(path, glob='**/*')[:1]
cfg.browser = browser
yield from browser_new.index()
Having to set up both promnesia and HPI backends considerably adds to the complexity of the setup.
IMO it'd be a considerable improvement if I could just pass a relevant configuration to HPI sources directly from Promnesia config.
So if I'm configuring Roam:
would be something like
And then a user does not need to figure out how to set up HPI config which is literally just used to provide that export path string