Closed tokejepsen closed 6 years ago
There are currently two issues to tackle:
mongod
does not get execute when running the container, because start_zou.sh
is blocking.admin@example
and password: default
.Whop! There's a lot to condense here. Off the top of my head:
Mongo can be run as a daemon with --fork
. It requires to log activity to a file, which is currently /avalon/mongo.log
.
I whipped up a script to sync CGWire -> Avalon.
import os
import gazu
from avalon import io as avalon
# Note: global..
gazu.client.set_host("http://192.168.99.100/api")
# Note: plain-text..
gazu.log_in("admin@example.com", "default")
print("Logged in..")
projects = []
objects = []
for project in gazu.project.all_projects():
assets = gazu.asset.all_assets_for_project(project)
shots = gazu.shot.all_shots_for_project(project)
for assets, silo in ((assets, "assets"), (shots, "shots")):
for asset in assets:
objects.append({
"schema": "avalon-core:asset-2.0",
"name": asset["name"].replace(" ", ""), # remove spaces
"silo": silo,
"data": {},
"type": "asset",
"parent": project["name"],
})
projects.append({
"schema": "avalon-core:project-2.0",
"type": "project",
"name": project["name"],
"data": {},
"parent": None,
"config": {
"schema": "avalon-core:config-1.0",
"apps": [],
"tasks": [
{"name": task["name"]}
for task in gazu.task.all_task_types()
],
"template": {
"work":
"{root}/{project}/{silo}/{asset}/work/"
"{task}/{app}",
"publish":
"{root}/{project}/{silo}/{asset}/publish/"
"{subset}/v{version:0>3}/{subset}.{representation}"
}
}
})
print("%d projects" % len(projects))
print("%d assets" % len(objects))
os.environ["AVALON_PROJECTS"] = r""
os.environ["AVALON_PROJECT"] = "temp"
os.environ["AVALON_ASSET"] = "bruce"
os.environ["AVALON_SILO"] = "assets"
os.environ["AVALON_CONFIG"] = "polly"
os.environ["AVALON_MONGO"] = "mongodb://192.168.99.100:27017"
existing_projects = {}
existing_assets = {}
installed_projects = []
print("Fetching Avalon data..")
avalon.install()
for project in avalon.projects():
existing_projects[project["name"]] = project
for asset in avalon.find({"type": "asset"}):
existing_assets[asset["name"]] = asset
print("Synchronising..")
for project in projects:
if project["name"] in existing_projects:
continue
print("Installing project: %s" % project["name"])
os.environ["AVALON_PROJECT"] = project["name"]
avalon.uninstall()
avalon.install()
avalon.insert_one(project)
for asset in objects:
if asset["name"] in existing_assets:
continue
asset["parent"] = avalon.locate([asset["parent"]])
print("Installing asset: %s" % asset["name"])
avalon.insert_one(asset)
Here's what I'm thinking.
This should help us prove the concept, before going much further.
Writing this, it became rather clear that our io.py
is ill devised. It's already showing legacy tendencies from the days before Launcher was made, such as expecting the environment to be fully qualified with project and all before being used. For a rainy day, we could have a look at breaking that dependency and making it into a more generic database browsing utility which it was ultimately made to be.
Very cool.
Would be good to have a centralized api or similar that needs implementing for other project managers, so it easier to know what needs doing. Would the io.py
be the place for this?
Yes, io.py
is the gateway to the database. inventory.py
builds on top of that, and is effectively a higher level version of io.py
, with an understanding of assets and things.
We should pop up a separate "refactoring PR" about it, but in a nutshell, one of the higher level goals of io.py
was to avoid having to explicitly reference the server and project whenever accessing the database from within a host.
That is, I wanted:
from avalon import api
for asset in api.ls():
print(asset) # list assets from current project
As opposed to..
from avalon import api
client = api.Client("mongodb://192.168.99.100:27017")
db = client["avalon"][api.current_project()]
for asset in api.ls(db):
print(asset)
The cost of the convenience however is more "under the hood" stuff, like having a project and db address set in the environment, upfront, which in retrospect complicates other aspects like what we're trying to do right now.
One more thing about the synchronisation.
Initially I was thinking that maybe it'd be worth switching Avalon to using the CGWire database entirely; skip the synchronisation step. But having interacted with it, on the surface there are a few problems with that.
import timeit
num = 100
dur = timeit.timeit(lambda: avalon.find({"type": "asset"}), number=num)
print("%.3f/sec" % (num / dur))
TLDR; we should stick to Mongo internally.
Currently cant get a working version of this, not even to the point of getting the Kitsu website up. Running into this issue:
Step 30/34 : RUN echo Initialising Zou... && /opt/zou/init_zou.sh
---> Running in f9452a43292f
/bin/sh: 1: /opt/zou/init_zou.sh: not found
But the file /opt/zou/init_zou.sh
clearly is present and copied just a couple of lines earlier.
Interestingly I had to switch the line-endings on init_zou.sh
to Unix style from Windows style endings.
I assume this is because I'm developing on Windows and when cloning the repository, Git assume Windows style endings.
First working dockerfile. Woop woop!
Apart from the line endings problem, there is also an issue when creating the default admin user with zou
. The email is being checked for validity here, so it can't be the example admin@example.com
.
Don't know how CGWire did this, with that email.
I have managed to remove all but two external files. I didn't remove nginx.conf
and supervisord.conf
because they are quite long, and I could not figure out a way of creating the files in the Dockerfile.
I'm also not entirely happy about the syntax of creation of the files, because its a lot of echo
run commands. Also it can probably be improved for readability.
Lastly I have temporary disabled the creation of the admin@example.com
user because of this.
For starters we should either get CGWire out of supervisord, which it is using for running services in the background, or we get Mongo and Samba into supervisord.
It'll probably very easy to get Mongo and Samba in supervisord. Do you mean we'll use supervisor to reduce the entrypoint to supervisord -c /etc/supervisord.conf
?
It'll probably very easy to get Mongo and Samba in supervisord. Do you mean we'll use supervisor to reduce the entrypoint to supervisord -c /etc/supervisord.conf ?
Yeah, seems reasonable I think.
That is Mongo and Samba running in supervisor. supervisor is actually quite a neat framework.
Dunno what do about the existing external files and the amount of RUN commands.
Should this PR be with the CGWire sync script as well? Imagining this to be run through supervisor.
Should this PR be with the CGWire sync script as well?
I think we can make that an independent PR. Think we have a few things to work out that doesn't involve getting CGWire up and running.
Imagining this to be run through supervisor.
Yeah, I think a plain Python process running on an infinite while loop should suffice, running some function and sleeping for 10 seconds.
Then we can open up a dialog with Frank about what we need from CGWire in terms of callbacks to get rid of it.
Dunno what do about the existing external files and the amount of RUN commands.
I'll do a pass over it now, see what I can do.
This is looking good to me. Think we still need to figure out the email issue before we can merge.
Ok, works!
$ docker run \
--name avalon \
-e AVALON_USERNAME=avalon \
-e AVALON_EMAIL=avalon@getavalon.github.io \
-e AVALON_PASSWORD=default \
-v avalon-db:/data/db \
--rm -ti \
-p 445:445 \
-p 27017:27017 \
-p 80:80 avalon/docker:0.4
Where the -e
are optional.
Nicely done! I'm pretty happy with this. Merge?
Done! Would you like to give the multi-container approach a try next?
Sure, lets
Oh no, I just realised something. The email is currently hardcoded into the image, as it's being installed on build, not on run.
Something to keep in mind for the multi approach.
That is a good point. Maybe we should have it as part of the entry points, that it'll make an admin account if an email is passed in?
Even more of a reason for splitting the tracking container from the rest. Going to be very specific behaviour for Ftrack and friends.
The email is currently hardcoded into the image, as it's being installed on build, not on run.
Think the approach will be that admin@example.com
will be the default admin user, and from that account users will create other admin accounts with more secure login, then delete the admin@example.com
account.
This is a PR to include CGWire into Avalon's distribution.