Add system overview diagrams.

[faiza-a]

@manthey please have a look at the diagrams in 'what is DSA section' and the 'system-overview-page.html'. Let me know if you want any changes in diagrams/caption/text/sizes etc. (I have put dummy text in description for now.) Also do you think its a good idea to add a link to 'system-overview' page in header?

[faiza-a]

@jtomeck In my last commit (b801a98855b87d95d43b7954bda604a876a971ec) I have created a copy of 'default.html' and modified it a little to create 'default-full-width.html' (to use it for system-overview page) which we can use for any pages on this site that we want to have in full width. Do you think this is right approach? can you have a look at b801a98855b87d95d43b7954bda604a876a971ec whenever you get time. and suggest any fixes or different approach.

[jtomeck]

Do you think this is right approach? can you have a look at b801a98 whenever you get time. and suggest any fixes or different approach.

I'm not sure what the new layout is giving you beyond the original default+page layout. It looks like you're still using a section with a .container inside of it, which is what the original page layout gives you. I'll make a PR to show you what I'm talking about.

Check out #105

[faiza-a]

Do you think this is right approach? can you have a look at b801a98 whenever you get time. and suggest any fixes or different approach.

I'm not sure what the new layout is giving you beyond the original default+page layout. It looks like you're still using a section with a .container inside of it, which is what the original page layout gives you. I'll make a PR to show you what I'm talking about.

Check out #105

The only change i did to default.html is to remove the section and container. so that it don't limit me to the container width. (i wanted to color whole page grey)

[manthey]

Here is a first pass for content for the system overview. The system overview could have an introduction paragraph:

(intro)

The Digital Slide Archive is typically either installed on a single computer or on one main computer with on or more worker machines to run image algorithms. There are additional options, such as having the database hosted in an external location, but these are uncommon.

Single Computer

The default installation of the Digital Slide Archive is on a single computer. This is the simplest was to deploy the DSA, though image analysis tasks are limited to the available processing power a single computer.

Image and other data files, the database, and log files are all stored on the local file system. Data files can also be stored or imported from external assetstores, such as Amazon S3.

When the deploy_docker.py script is used for installation, a set of five docker containers are started to provide all of the services needed by the DSA. When an image analysis task is performed, e.g., detecting nuclei on a whole slide image using the HistomicsTK toolkit, an additional docker container is created to run the task. This container only exists as long as is necessary for the image analysis. The results are stored back to the database and file assetstore.

Distributed Workers

To allow running a larger number of image analysis tasks, the Digital Slide Archive can be installed on one main computer and any number of worker machines.

Image and other data files, the database, and log files are all stored on the local file system of the main computer, just like in the single computer configuration. Of course data files can also be stored or imported from external assetstores, such as Amazon S3.

To start in this mode, some additional parameters are passed to the deploy_docker.py script. For instance, a minimal command is deploy_docker.py start --only girder,rabbitmq,mongodb,memcached --worker-api-url=<hostname or IP of this computer as seen from second computer>/api/v1. One or more workers can be started on other machines with a command like deploy_docker.py start --only worker --rmq=<hostname or IP of first computer>. Workers can be started or stopped at any time. For instance, more workers can be added when running many tasks and then stopped when they are no longer needed. If a shared network file system is used, appropriate mount commands can be added as additional options to both the main computer and the worker commands; this can substantially reduce network traffic.

On the main computer, four docker containers are started to provide most of the services needed by the DSA. On each worker, one docker container is started initially. When an image analysis task is performed an additional docker container is created on the worker to run the task. This container only exists as long as is necessary for the image analysis. The results are stored back to the database and file assetstore.

When there are multiple workers, each worker is given tasks in turn until they are all busy. As workers finish a task, they are assigned new tasks. If a worker is stopped or loses network connectivity with a partially completed task, that task is reassigned to another available worker. The worker computers do not need to be identical; faster workers will end up processing more tasks than slower workers.

[cooperlab]

@faiza-a I'd like to make a pass on @manthey comment text when you get this integrated with docs/system-overview/index.md.

We have a lot of image analysis scenarios that can be confusing: 1. Running jobs through the API or HistomicsUI on the systems where DSA is hosted (and workers) 2. Running local jobs using the API and 3. Running local jobs using HistomicsTK as a stand-alone package. I won't elaborate on all of these in that doc but want to be explicit that we are talking about 1.

[faiza-a]

@faiza-a I'd like to make a pass on @manthey comment text when you get this integrated with docs/system-overview/index.md.

We have a lot of image analysis scenarios that can be confusing: 1. Running jobs through the API or HistomicsUI on the systems where DSA is hosted (and workers) 2. Running local jobs using the API and 3. Running local jobs using HistomicsTK as a stand-alone package. I won't elaborate on all of these in that doc but want to be explicit that we are talking about 1.

@cooperlab i have integrated the descriptions in docs/system-overview/index.md. have a look at the web page and let me know any changes/additions you want in it.

[manthey]

@faiza-a I push a small commit to change how the commands I had in my sample text are rendered (and to make the diagram links relative). Usually the background of formatted commands is somewhat different (like in the Github comment), but I don't perceive any contrast on the background on the system diagram page. We may want to adjust the css for that.

[manthey]

Can we make the clicking on the diagram on the main page go to the system overview page (besides having the link below it)?

[faiza-a]

@faiza-a I push a small commit to change how the commands I had in my sample text are rendered (and to make the diagram links relative). Usually the background of formatted commands is somewhat different (like in the Github comment), but I don't perceive any contrast on the background on the system diagram page. We may want to adjust the css for that.

Thanks. (i somehow missed your comment above to change to relative link). The formatted text looks good. i'll adjust the text background.

[faiza-a]

Can we make the clicking on the diagram on the main page go to the system overview page (besides having the link below it)?

Yup ill do it

[faiza-a]

Can we make the clicking on the diagram on the main page go to the system overview page (besides having the link below it)?

Yup ill do it

Fixed in 56d6397

[faiza-a]

@faiza-a I push a small commit to change how the commands I had in my sample text are rendered (and to make the diagram links relative). Usually the background of formatted commands is somewhat different (like in the Github comment), but I don't perceive any contrast on the background on the system diagram page. We may want to adjust the css for that.

Thanks. (i somehow missed your comment above to change to relative link). The formatted text looks good. i'll adjust the text background.

Fixed in 0ac77a4

[manthey]

@cooperlab I recommend we merge this and then add any changes you want as a separate PR. If you'd rather make the changes to this PR, let us know.

[cooperlab]

I made a commit on the system diagram description.

On Mon, Apr 20, 2020 at 9:19 AM David Manthey notifications@github.com wrote:

@cooperlab https://github.com/cooperlab I recommend we merge this and then add any changes you want as a separate PR. If you'd rather make the changes to this PR, let us know.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/DigitalSlideArchive/digital_slide_archive/pull/103#issuecomment-616585060, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAK65EMFGQE6OQOUW3L2D6LRNRKYPANCNFSM4MHCQASQ .

[manthey]

@cooperlab Thanks. A definite improvement.

[faiza-a]

Thanks @cooperlab @manthey. i'll merge this PR as soon as checks completes.