Closed ahundt closed 4 years ago
Good idea. Let me know if you need help with any of these.
Any news? Can I help with this?
Not much progress unfortunately. Yeah that would be great, unfortunately I've ended up just doing regular json dumps to a folder for my own work because I ran out of time to spare on this, and simplified setup + creating ways of calling sacred from existing apis without using the experiment @ tag setup still seem to be in progress.
@ahundt and @uatach : You are interested in a full example on how to use MongoDB in the docs, right? This should not be hard to do.
On Ubuntu sudo apt install mongo
installs a local MongoDB (though official docs recommend using their mongodb-org package).
You might need to manually start the demon with sudo service mongodb start
(not sure).
Of course the docker based setup should also work, but might require some network configuration and later specifying the url in the mongoobserver.
You can verify if it is running correctly by starting the mongo client with mongo
.
With that in place you can just use -m my_sacred_db_name
on your experiment to add a MongoObserver
that writes to that database.
Make sure to have the pymongo
package installed (pip install pymongo
)
You can use the mongo client to look at stored results.
$ mongo
MongoDB shell version v3.4.7
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.4.7
> show dbs
admin 0.000GB
local 0.000GB
my_sacred_db_name
> use my_sacred_db_name
> show collections
fs.chunks
fs.files
runs
> db.runs.findOne()
{
"_id" : 1,
"experiment" : {
"name" : "hello_cs",
"base_dir" : "/home/greff/programming/sacred/examples",
....
But maybe a nicer way is to query from python using pymongo:
import pymongo
client = pymongo.MongoClient()
runs = client.my_sacred_db_name.runs
entry = runs.find_one()
Which returns a dictionary with the stored results for a run. The mongo interface has rich capacities for querying, to find the right experiment entry.
Calling the experiment with the retrieved config:
from my_experiment import ex
ex.run(config_updates=entry['config'])
Is that the information you were looking for? @ahundt : What do you mean by "creating ways of calling sacred from existing apis without using the experiment @ tag setup still seem to be in progress."? @uatach : If you clean up these instructions I'd be happy to include them in the docs.
I mean something very similar to the stuff that was discussed in https://github.com/IDSIA/sacred/issues/193, I have a bunch of code where I could easily pass something object oriented in and add to it, but that code doesn't mix well with how experiments are defined in a global namespace, and I didn't find examples using sacred configs and variables within the context of a class or another function that fit well with my existing code base. It is certainly possible I missed something obvious, I just ran out of time to figure it all out.
For example, I'm using gpyopt for hyper parameter optimization, so if that chooses my config ~2000 times in a single hyperopt run which is passed in a callback, so I wasn't sure about correctly storing it in sacred without breaking anything.
Ultimately I started dumping a dictionary to json was a one liner json.dump()
, I threw a time stamp into the filename and that covered what was truly essential to my project with lower time investment. I know that obviously lacks many of the nice sacred features, but I've been able to reproduce my results reasonably well with that approach. Heh, sorry if that sounds less than ideal, I just had to move along quickly.
@ahundt No worries! Feel free to use Sacred any way you like. ;-) I was just trying to figure out if there was anything important left for me to do.
@Qwlouse That mongo example would be great in the documentation. It would be even better if there was a good example of how to link this was sacredboard(though I know this is outside the scope of this repo)
The desired setup instructions should be documented at https://sacred.readthedocs.io/en/stable/examples.html#docker-setup. For a complete walk trough see also https://github.com/JarnoRFB/pydata_berlin_sacred. Closing this for now. Feel free to reopen if you feel that something is missing.
Once #205 is resolved, can we add a section to the sacred docs with:
(I don't quite know how to do all of this yet)