Have Dynmap work with separate hosts with all features.

[ZacharyDuBois]

Could you add in the ability to have all of the features of the Dynmap plugins with separate hosts for the web server and Minecraft server. I use completely separate hosting companies and want to have dynmap upload the tiles upon them updating to my web server that can host the interface. I also want to be able to have the live player stats and stuff. I think that website stuff should stay with web servers and Minecraft stays with Minecraft servers.

• Thanks -- Zach

[ajfite]

This is already possible if you have some way of pushing the map tiles from the Minecraft server host to the web server host, look up the standalone web server configuration instructions.

[rpgmoods]

I am having trouble trying to accomplish this as well. The standalone instructions are for a web server on the same machine (unless i just missed it) and i have tried to adapt it to my needs but I got stuck. Maybe i have not looked at the right instruction page. Could you possibly link the configuration instruction you speak of. I would love to get this working

[ZacharyDuBois]

Yeah. You need a section on how to do this. Or add this in to the plugin if it does not exist.

[ZacharyDuBois]

Could anyone review this feature/issue/documentation request?

[frymaster]

check out the custom-commands -> postupdatecommand node in the config. This allows you to run an arbitrary command upon tile update. This could either be a direct file transfer command, or something that adds the file to a list of updated files to be transferred in bulk at a later time.

Apart from that, you should be transferring the JSON files generated by the plugin to the webserver, and vice versa, again on whatever schedule suits you best

[bezeek]

Unless you're trying to use features I'm not aware of, there's nothing preventing any features from working just fine since v1.2 (and perhaps earlier.)

The short answer is: use NFS.

I can't get into too much detail because it's been a while since I set it up, but basically you want to make an NFS connection between the two hosts (I have the web dir on my web host, mounted to my MC host.) Start by following the standalone instructions, set up the NFS connection, and configure permissions on both servers so your MC host's Java process can write to the NFS mount. Additionally, you'll want to edit standalone/dynmap_access.php on the web host so it points to a path that your HTTP server can understand, and make that file read-only so Dynmap doesn't update it.

If none of the above paragraph makes sense, you're in over your head and should read up on NFS a bit before proceeding.

Lastly, a few notes: • It would appear that v1.5 (maybe even v1.4) might offer a better way to accomplish this, but I've yet to find a reason to bother messing with success. • If you experience performance issues, read more about NFS, and poke around in Dynmap's configuration.txt to reduce the frequency of data writes. • NFS is naturally unencrypted! If this is an issue for you, I suggest contacting your host(s) to explore secure (IPsec or OpenVPN) tunneling options. If possible, move both services to one host which supports intranet VLANs. • You might run into an issue where Dynmap refreshes constantly. If this happens, simply do a "/dynmap reload" over and over until this ceases. Give it a few seconds between each reload to let it settle, and don't be surprised if it takes up to 10 attempts for the refreshing to stop.

(Somewhere buried in this Github project, it is stated that the reason this isn't documented is because the way you go about such a thing varies depending on your needs, your operating system, your knowledge of systems administration, and what your hosting company/companies allow. I might add, an advanced configuration such as this probably shouldn't be attempted by anyone who needs to ask how it's done - you could easily break one or both of your servers, and/or create gaping security holes in the process!)

[ZacharyDuBois]

Well these would be useful tips on how to develop a way to do this. I would not always trust that process however. But I hope this feature is implemented soon. My hosting company told me this when I asked for a space increase, "You currently have a 30 GB limit and only 7.2GB of that data is for worlds. Unlimited storage only applies to the game server world data. Additional data, such as backups or files generated by third party plugins, are not eligible for storage limit increases." :/

[bezeek]

NFS is a beast of a protocol developed over nearly 30 years, requires a handful of ports to be open and unrestricted, and would frankly be absurd to implement within a Bukkit plugin.

Also, it would allow dynmap to save all of its data directly to your web host's hard drives. thereby requiring no extra space on your MC host's drives. (As mentioned in my last comment, I have my Bukkit configured to mount and save to my web server, partly for performance reasons, and partly to keep the load off Bukkit's hard drives.)

I wouldn't get my hopes up, but perhaps you can ask your Minecraft host (very, very nicely) if you may mount a remote NFS share. If you have SSH access, you can probably do it yourself; NFS support is almost universally included in Linux distributions.

[frymaster]

The lesson you need to take home is no one is going to hand you a solution on a platter. The way bezeek would do it requires a fast connection between the servers, and a lot of bandwidth. The way I'd do it does not, but requires a local copy of the files(which won't help if you are running low on local storage. What tools you have depends on if the servers are Linux or windows etc.

[bezeek]

Very nicely stated. That said, a dedicated server is the only surefire way to get the job done.

@ZacharyDuBois: If you want to pursue additional options, i.e. help locating a host that suits your needs, feel free to shoot me an email. The address is visible on my GitHub profile.

[rbos]

If I were going to do this, over the Internet, I would use a snapshot filesystem with the ability to export/import filesystem diffs. ZFS, for instance. Then, you can periodically squirt those diffs, compressed, over the Internet and import them onto your second host. Then, you can render the map on the main host, and copy the tiles over as needed, or, alternatively, mirror the game files to the second host and process the map there somehow.

[pr4xt3roy]

I do not see why NFS would be required (or any export/import filesystem). Files have to go only ONE way. They have to be overwritten on the host where dynmap is located. They do not have to be read. No locking system is required. No attributes are required. All things that NFS and friends suports which we do not need. We just plain and simple writes. Am I missing something? Maybe failover... If the destination host is down, NFS will get them over once the dynmap host is back up (assuming the NFS is exported from the minecraft server and then mounted on the dynmap host).

I am thinking that a very simple REST application running (So small and simple it could be called an applet) on the host where dynmap is which receives a PUT request with the information about the filename of the tile and the content. The REST application would simply received and write it on the disk. Using the postupdatecommand config you can simply use curl (on unix) to send the updated file using PUT on the other server.

I am using AWS which supports queues. So to support failover, all I need is to send the tiles to the AWS queues and have the dynmap host poll the queue and update the tiles as they arrive, one by one. In this case the postupdatecommand would push the tile in the AWS queue.

Thoughts?

[frymaster]

NFS or similar would be required for the original creator of this bug because he requires this feature mainly because his minecraft server host does not have enough space for the map tiles.

For situations where that's not needed, something like you suggest would work well

[pr4xt3roy]

I see. Then if the NFS is exported from a the dynmap host, there could be failure to write tiles when the dynmap host would be down. The mount would be missing. It could even cause some java thread to hang, waiting for the mount to respond back. I wonder if the whole Minecraft server would hang. I've seen that happen in other kind of application relying on NFS. NFS does have some benefits, but I tend to stay away from it. Too much trouble to handle failure in some cases.

If the host have enough memory, would it make sense to create a ramdisk and put the tiles on the ramdisk? When the server reboot, you'd loose all your tiles, but maybe some people would not mind!?!?

Just a thought!

[mikeprimm]

Two points:

1) Tile files don't only go one way: when a new tile is rendered, the previous zoomed out tiles 'above' it need to be updated - which either requires reading the older zoomed-out tile (to update the corner that was rendered) or reading the potentially other 3 tiles contributing to the zoomed out tile (to regenerate the whole thing). Either way, tile processing is read-modiify-write for all zoom out processing

2) Loss of coherence is a huge issue: if we are not able to write a tile update when we have produce it, there is no reasonable way to re-establish coherence (short of a full rerender). This can be addressed by at least remembering which tiles failed to be updated (that is, keep them marked as 'dirty', and try rendering and publishing them again later). The 'hashcodes' are also used for this sort of purpose (since many to most tile updates do not result in changes in the visual content of the tiles - picture how often folks are mining below the surface, yielding block changes that result in no visible update) : hashcode coherence with the published tiles is critical, since we avoid overwriting files and pushing updates to clients based on hash code matches.

The zoom out processing (a fundamental part of actually having a usable map) mandates that there be read/modify/write processing of the tile data. It could be done on the remote end, assuming that local processing and/or scripting was assumed (e.g. push the base tile to the remote end, and let the script there cook up the zoom out tiles), but for folks wanting to use crap hosting services that just have FTP access and no local scripting or shell access, this doesn't work - and even for folks that do, its a support nightmare (that is, I'm not talking some fraction of 25k user servers through the process of setting up a web host I don't have and they don't know how to administer....).

Rendering and maintaining on a local file system, and driving incremental publishing to a remote file system, is quite workable - but doesn't help the folks that are complaining about space or file count: actually makes it worse.

[bezeek]

@pr4xt3roy, it does in fact cause the thread to hang and the server eventually crashes. For people who need a canned solution but also can't guarantee 100% uptime of their web server, iSCSI could be a viable alternative.

[ZacharyDuBois]

Lol. I was laughing when I saw this in my inbox. Some Amazon S3 + CloudFront would be great. Very cheap and reliable.

[ZacharyDuBois]

Heck if there is S3 and CloudFront support, GIVE A NEW SETTING! Ultra High Res!

[TomLewis]

sorry to dig up a 2 year old topic, but most VPS's are running on small SSD's now, and were in need of a solution to save tiles etc on external servers, Im down for S3 & CloudFront.

Hell, Could I mount an S3 bucket on Debian and symlink the folder?

[pr4xt3roy]

@FrozenBeard If you are using AWS, could this be a workable solution? https://aws.amazon.com/blogs/aws/amazon-elastic-file-system-shared-file-storage-for-amazon-ec2/

[TomLewis]

@pr4xt3roy Im not using any AWS as of yet, I have a set of servers on a OVH VPS and my main server is still on a shared host, that Dynmap has a symlink to a HDD, but my tiny 40GB OVH SSD VPS Im trying to find a solution to move the Dynmap tiles to another location, I have many shared web-servers I could use if I could mount them, Never used AWS before, but im also looking into them as a backup solution too?

[ZacharyDuBois]

You could NFS it. That's what I did when I used this plugin. Initial render takes forever though. I use MC overviewer now.

[mikeprimm]

Actually, since this was first opened, I've added support for map data to be stored in alternate places, based around an in-code storage driver (currently, there are drivers for 'normal' file system storage, MySQL database, and SQLite database). I'll give a look at how practical doing an S3 driver would be, but I suspect it may not be that bad.

[TomLewis]

@mikeprimm You can STORE the tile information in MySQL? Holy crap! I thought this was for settings or saving player movement information, I had no idea you can store images in MySQL! What kind of sizes are we looking at for databases now? I would imagine they are absolutely gigantic?

[mikeprimm]

The dominant space usage is pretty similar to file system - some improvement due to fragmentation overhead (e.g. file systems allocate space in chunks, not bytes), with some overhead in the DB encoding itself. I haven't done a scientific A vs B comparison, but I'd wager its within +-10% of the same size, for a given map. The upside is the DB server doesn't need to be the same box as the MC server, nor the web server.

[TomLewis]

I'm definitely going to give this a shot with my crappy shared hosting, After I request an external IP connect. Does this have any downsides? Like increased stress on the network/MySQL server? Does it do a tremendous amount of read query's when using the map?

On 11 September 2015 at 19:46, mikeprimm notifications@github.com wrote:

The dominant space usage is pretty similar to file system - some improvement due to fragmentation overhead (e.g. file systems allocate space in chunks, not bytes), with some overhead in the DB encoding itself. I haven't done a scientific A vs B comparison, but I'd wager its within +-10% of the same size, for a given map. The upside is the DB server doesn't need to be the same box as the MC server, nor the web server.

— Reply to this email directly or view it on GitHub https://github.com/webbukkit/dynmap/issues/1135#issuecomment-139624615.

[Psy-Virus]

TL:DR

If still needed, pls reopen. Or better, create a new one ^^