tatornator12 commented 7 years ago

Thank you for the great instructions! I'm very new to Docker and have only been able to spin up containers with postgresql, so this is new territory and am unsure if this is an issue.

I am able to replicate your workflow seamlessly until I try to run the container in either detached or interactive mode. I keep getting this error response:

Error response from daemon: driver failed programming external connectivity on endpoint server (708a2c7db5137e56a8d7b022dd00522ca4d9ed5265252be52e6f6e22726ebb60): Error starting userland proxy: Bind for 0.0.0.0:6443: unexpected error Permission denied.

I checked Task Manager to see what services were using that Local Address and it seems that javaw.exe from C:\Program Files\ArcGIS\framework\runtime is occupying that space. Killing the process won't fix the issue. Any thoughts on this?

tatornator12 commented 7 years ago

Okay, I decided to try running it in Interactive Mode again and changing the ports to 6081:6080 and 6444:6443. Within the bash shell, I've tried using the server/startserver.sh and then I run into another error:

/home/arcgis/server/framework/etc/tools/set_temp_directory: 27: .: Can't open /home/arcgis/server/usr/init_user_param.sh

I checked that location and the file is in fact missing when it gets installed.

tatornator12 commented 7 years ago

Alright, I went ahead and grabbed the init_user_param.sh file from a separate install of AGS on Linux and moved into the /arcgis/server/usr directory. I've successfully been able to access the setup page on Server Manager, but am faced with yet another issue:

untitled

brian32768 commented 7 years ago

I am here and listening but unable to jump on this until the weekend. I am glad you are trying it out.

Brian

On Fri, Aug 25, 2017 at 12:21 PM, Taylor Teske notifications@github.com wrote:

Alright, I went ahead and grabbed the init_user_param.sh file from a separate install of AGS on Linux and moved into the /arcgis/server/usr directory. I've successfully been able to access the setup page on Server Manager, but am faced with yet another issue:

[image: untitled] https://user-images.githubusercontent.com/23561673/29729168-090b8e00-89a9-11e7-9b9b-20e947513743.png

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Geo-CEG/docker-arcgis-enterprise/issues/2#issuecomment-325013728, or mute the thread https://github.com/notifications/unsubscribe-auth/AB_oxKhIkB6MkX1n4ts1bnzTDO1O7i5pks5sbx7GgaJpZM4O-0Xh .

-- Brian Wilson tel:707-827-0001

tatornator12 commented 7 years ago

No worries! We all have day jobs :) I'm just testing your walkthrough as part of mine. I won't be able to return to it until next week. Troubleshooting through the issues has really helped me learn Docker more. If you have any advice on the issues I'm facing, I'm all ears.

Thanks again for your work!

tatornator12 commented 7 years ago

Any thoughts on how to proceed with this one? I used Fiddler to check why the specific address is unreachable and received a DNS error:

DNS Lookup for "11833823f4bd" failed. System.Net.Sockets.SocketException No such host is known

brian32768 commented 7 years ago

Pretty new at it myself, I am building this to learn. Back working on it today.

On Mon, Aug 28, 2017 at 4:49 AM, Taylor Teske notifications@github.com wrote:

Any thoughts on how to proceed with this one? I used Fiddler to check why the specific address is unreachable and received a DNS error:

DNS Lookup for "11833823f4bd" failed. System.Net.Sockets.SocketException No such host is known

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Geo-CEG/docker-arcgis-enterprise/issues/2#issuecomment-325332125, or mute the thread https://github.com/notifications/unsubscribe-auth/AB_oxKyi0L0dLMtR_A06BmP9Ikwn76PAks5scqlMgaJpZM4O-0Xh .

-- Brian Wilson tel:707-827-0001

brian32768 commented 7 years ago

I just wrote more documentation now. ESRI's component seem to want to know their own hostnames, and get upset if the names change. This is annoying in Docker world. There are scripts in each folder to start each component, so for example in arcgis-server there us "runags" and in portal-for-arcgis there is "runportal".

You change into the directory and run the script (./runags) and when the script runs it gives you some info and drops to a command prompt. You type ./start.sh and off it goes. If you don't want to deal with the interactive command line session you can run the "start" script for example "./startags" for ArcGIS Server. This will start the container in detached mode and automatically run the start.sh script. The reason to use runags is to see what's going on in the container.

Even if you don't want to use these scripts they will show you the command line arguments I am using. I am trying to persist data on the local file system with volumes, so far unsuccessfully. ArcGIS is just not cooperating yet. The problem you describe (the DNS name lookup fails) is because you are not using the hostname, net, and net-alias options. This is a big hack. In a separate DNS server running dnsmasq I have this

 # NB The numbers are dependent on the order you start the dockers
 172.18.0.2 server server.arcgis.net
 172.18.0.3 portal portal.arcgis.net
 172.18.0.4 web-adaptor web-adaptor.arcgis.net
 172.18.0.5 datastore datastore.arcgis.net

I was hoping to move beyond this before someone else started hacking around with this. :-) Let me know if this is a big obstacle for you, I will give it more attention.

You can repeat starting the other components (portal, web-adaptor, datastore). I normally use a separate window for each so I can watch and interact which each component. As you bring each component up you can connect to it via browser; the script gives you the URL to use.

I have only a vague sense of how these components tie together, and am especially hazy on what datastore is for. I am learning docker and arcgis enterprise at the same time.

brian32768 commented 7 years ago

I started using the wiki up there on Github.

https://github.com/Geo-CEG/docker-arcgis-enterprise/wiki

On Mon, Aug 28, 2017 at 4:49 AM, Taylor Teske notifications@github.com wrote:

Any thoughts on how to proceed with this one? I used Fiddler to check why the specific address is unreachable and received a DNS error:

DNS Lookup for "11833823f4bd" failed. System.Net.Sockets.SocketException No such host is known

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Geo-CEG/docker-arcgis-enterprise/issues/2#issuecomment-325332125, or mute the thread https://github.com/notifications/unsubscribe-auth/AB_oxKyi0L0dLMtR_A06BmP9Ikwn76PAks5scqlMgaJpZM4O-0Xh .

-- Brian Wilson tel:707-827-0001

tatornator12 commented 7 years ago

Great work! And thanks! I'll try to take a look at it today.

Data Store is an application that lets you easily configure data storage for hosting and federated servers used with your portal. Under the hood, it uses postgresql.

tatornator12 commented 7 years ago

So I didn't do much differently, but I was finally able to proceed with creating the site for Server. I started from scratch:

Created the network arcgis.net
Built the dockerfile
Ran the script (./runags) within Git Bash
Started the Docker container in interactive mode on the correct ports this time (6080 & 6443). Previously I had a local version of the ArcGIS Enterprise installed on my computer for testing purposes, so I stopped those services completely before proceeding with this setup. For some reason, starting it in interactive mode only works for me at this point.
Authorized the software
Directly ran the startserver.sh file (rather than use the ./start.sh script)
Proceeded to https://localhost:6443/arcgis/manager and was able to successfully create the site.

Nevertheless, I don't know if this is a completely successful attempt since I can only access the site through localhost (or my local machine name) rather than through https://arcgis.localdomain:6443/arcgis/manager (using the docker container ID) as you had documented.

I plan on moving on tinkering with setting up the rest of the Enterprise, as you have documented. I'll let you know how that goes. I assume your ultimate plan is to automate the entire process, correct? I don't know if that's possible with Docker or if it would have to be done with a combination of Ansible and Docker.

Thanks again for your help with this one!

PS - I should note that I'm doing all of this on the Windows version of Docker.

tatornator12 commented 7 years ago

Found that the previous init_user_param.sh file was leftover from the previous install when I had to share my C drive. Still don't know why that file is not being created during install.

tatornator12 commented 7 years ago

So I prefer to install the platform in this order: (1) Server, (2) Data Store, (3) Portal, (4) Web Adaptor. I modified the your Dockerfile slightly for installing the Data Store, which included exposing port 9876 for a relational database, commenting out ADD federate.py, and commenting out ADD UPPER.py (this gave me an error during the build).

Interestingly, following a similar pattern of steps as above, I was able to access the Data Store setup page and proceed through the steps. When I clicked finish it ran and spit out an error (I don't remember what it was now), so I tried again with a fresh reset of Docker. Now every single time I try to finish the setup I get Method argument cannot be null. This is certainly different from the previous error. Any thoughts?

brian32768 commented 7 years ago

I am trying to put together a Compose file for this right now, I will change the order of the components. I got stuck when Compose failed on a path with a space in it "Program Files"! I will probably code around that and ignore it for the moment as I have already done many times on this project

On Wed, Aug 30, 2017 at 7:47 AM, Taylor Teske notifications@github.com wrote:

So I prefer to install the platform in this order: (1) Server, (2) Data Store, (3) Portal, (4) Web Adaptor. I modified the your Dockerfile slightly for installing the Data Store, which included exposing port 9876 for a relational database, commenting out ADD federate.py, and commenting out ADD UPPER.py (this gave me an error during the build).

Interestingly, following a similar pattern of steps as above, I was able to access the Data Store setup page and proceed through the steps. When I clicked finish it ran and spit out an error (I don't remember what it was now), so I tried again with a fresh reset of Docker. Now every single time I try to finish the setup I get Method argument cannot be null. This is certainly different from the previous error. Any thoughts?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/Geo-CEG/docker-arcgis-enterprise/issues/2#issuecomment-326014040, or mute the thread https://github.com/notifications/unsubscribe-auth/AB_oxFWTvZl4nuKCHSDlX96JYUol1-Ytks5sdXXmgaJpZM4O-0Xh .

-- Brian Wilson tel:707-827-0001

brian32768 commented 7 years ago

Will follow up on this more after breakfast,

I unpacked ArcGIS_Server_Linux_1051_156429.tar.gz which contains installers

ArcGISServer.tgz so then I unpacked ArcGISServer/setup_resources/ArcGISServer.tgz

Inside that tree I do a "find" and I see usr/init_user_param.sh

-----------------------------------------------------------------------

#

init_user_param.sh - User environment variable initialization script

#

This file is sourced by $installDir/framework/etc/arcenv as part of

the ArcGIS Server startup process. Any uncommented values will be

applied to the Server runtime environment.

#

You should only modify sections that are relevant to your configuration

and add user-defined variables at the bottom.

#

To enable the variable, remove the '#' from the beginning of the line

and replace the <...> with the appropriate values for your environment.

-----------------------------------------------------------------------

etc etc

One of the most interesting parts of this project is how much open source is embedded in AGE. I wonder about the efficiency of running several copies of PostgreSQL. Then again, I wonder about the efficiencies of Docker too. :-) RAM is not THAT cheap; I am currently finding I need more.

tatornator12 commented 7 years ago

Interesting....I wonder why it's not showing up in my install within a Docker container. When I installed it on a Linux VM that file showed up in the proper place....yeah AGE is quite a beast! I was actually just thinking about the RAM usages of Docker this morning. I was wondering if the RAM deficiency was directly related with my first error with setting up the Data Store. I have no idea what's going on with my new error. The logs haven't been much help either:

time="2017-08-30T15:16:42,283" type="SEVERE" code="110506" source="Data Store" process="90" thread="21" methodName="" machine="B9F635339459" user="" elapsed="">Attempt to configure data store failed. java.lang.NullPointerException: Method argument cannot be null.

Trying a fresh install of Docker now...wonder if I did something wrong....

tatornator12 commented 7 years ago

Alright, the fresh install seemed to do the trick. I'm now seeing the error I got yesterday. This is the error:

Could not connect to server on machine 'tteske'. ArcGIS Data Store or ArcGIS Server on that machine may not be running or the machine is not reachable at this time.

I noticed that when the Data Store container tries to make a connection to my Server container, the Server container becomes on unreachable and creates an unhealthy status when running docker ps. It seems the containers don't want to play nicely with each other.

Edit: It also seemed to have taken out the connection to the Data Store container. Edit2: Another interesting tidbit: if run the commands ./stopserver.sh and ./startserver.sh, I can get back into Server Manager and there is a Data Store registered. However, that Data Store cannot validate.

tatornator12 commented 7 years ago

Figured out that my underlying issue for the containers not communicating with each other comes back to the issue I mentioned above about only being able to access the sites using localhost rather than the Docker container ID or name. I pinged the addresses for the containers (172.18.0.2 & 172.18.0.3) and they were unreachable. I have a feeling this will be the biggest issue moving forward, so I'll have to determine how to resolve it now. Yay, Windows....

brian32768 commented 7 years ago

For me this project is predicated on running on Linux so I have not worked through any Docker-on-Windows scenario. Thinking a step further, I don't actually own any Windows machine right now! Mac, yes. Linux, yes. I run virtual machines on both of those to run ArcGIS.

I presume you've confirmed those really are the addresses and since they are on 172.18 you did create a separate Docker network, that part was easy for me. The ugly part happens when the ESRI code fights back and refuses to accept that the hostname can change. I have been working around that so far by nailing down hostnames and putting entries into DNS. I am hoping Docker Compose helps out with this.

On Wed, Aug 30, 2017 at 9:56 AM, Taylor Teske notifications@github.com wrote:

Figured out that my underlying issue for the containers not communicating with each other comes back to the issue I mentioned above about only being able to access the sites using localhost rather than the Docker container ID or name. I pinged the addresses for the containers (172.18.0.2 & 172.18.0.3) and they were unreachable. I have a feeling this will be the biggest issue moving forward, so I'll have to determine how to resolve it now. Yay, Windows....

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/Geo-CEG/docker-arcgis-enterprise/issues/2#issuecomment-326053296, or mute the thread https://github.com/notifications/unsubscribe-auth/AB_oxFAWbPtiLbRpchDmwveWf8Ce6viJks5sdZQ1gaJpZM4O-0Xh .

-- Brian Wilson tel:707-827-0001

tatornator12 commented 7 years ago

Haha! I primarily work on Windows, but also own a Mac for personal use. I haven't done much on Linux until recently. I wouldn't be surprised if many of my issues stem from being in a Windows environment.

[
    {
        "Name": "arcgis.net",
        "Id": "6504b106e029ab2e68d4688ce40244e44745b6bce7fc2ad7f6320d10526e4629",
        "Created": "2017-08-30T15:53:59.567818Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "5223209db849f84b3da3b49090e4a04a6137306dd3683010aa09c7c67cc5225d": {
                "Name": "ds_server",
                "EndpointID": "f0b9173c3524c6a1da85ce1e649cb84547913380bf4515a1af0589b72841531c",
                "MacAddress": "02:42:ac:12:00:03",
                "IPv4Address": "172.18.0.3/16",
                "IPv6Address": ""
            },
            "d6586a5c80921d10f9bfac57ad2813568acdf64fa6736f67275a2686a6806a18": {
                "Name": "server",
                "EndpointID": "ff167be61c8d252d90e186c4d319a298c356abfd5b3ce6375c656875ddb5731f",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]


Pinging 172.18.0.2 with 32 bytes of data:
Reply from 70.168.109.245: Destination net unreachable.
Reply from 70.168.109.245: Destination net unreachable.
Reply from 70.168.109.245: Destination net unreachable.
Reply from 70.168.109.245: Destination net unreachable.

Ping statistics for 172.18.0.2:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Yep, as you can see when I do docker network inspect arcgis.net. I also use the hostname, net, and net-alias options when running the containers. The hostname change is also a pain....

tatornator12 commented 7 years ago

I did some research on the issue and, to no surprise, it might be a Windows issue. Will have to look further into it.

Edit: This is what I see when I run ifconfig within the Server container:

eth0      Link encap:Ethernet  HWaddr 02:42:ac:12:00:02
          inet addr:172.18.0.2  Bcast:0.0.0.0  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:32190 errors:0 dropped:0 overruns:0 frame:0
          TX packets:29948 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:5225307 (5.2 MB)  TX bytes:23681677 (23.6 MB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:198168 errors:0 dropped:0 overruns:0 frame:0
          TX packets:198168 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:41976211 (41.9 MB)  TX bytes:41976211 (41.9 MB)

rewt commented 5 years ago

Getting this when browsing URL

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/arcgis/manager\"",
  "reason": "Forbidden",
  "details": {

  },
  "code": 403
}

Wildsong / docker-arcgis-enterprise

Unable to Run Container #2

-----------------------------------------------------------------------

init_user_param.sh - User environment variable initialization script

This file is sourced by $installDir/framework/etc/arcenv as part of

the ArcGIS Server startup process. Any uncommented values will be

applied to the Server runtime environment.

You should only modify sections that are relevant to your configuration

and add user-defined variables at the bottom.

To enable the variable, remove the '#' from the beginning of the line

and replace the <...> with the appropriate values for your environment.

-----------------------------------------------------------------------