search

bobafetthotmail / folder2ram

mount those folders to ram without losing access to their counterpart on disk!
GNU General Public License v3.0
110 stars 21 forks source link

Create dir2ram #1

Closed Reed97123 closed 8 years ago

Reed97123 commented 8 years ago

Hi,

Following on your good work, I've created a new version of folder2ram called dir2ram which extends the work to accomplish most of your todo list as well as fixing some of the bugs that I've experienced (long directory mangling, services needing to be stopped).

I was hoping you could give feedback on features and any bugs you encounter.

Summary:

Example Config file:

EXAMPLE:

Name FS MB LogHist Dependent_Service(s) Dir_To_Cache

smart tmpfs 8 5 smartd,smartmontools /var/lib/smartmontools

sickbeard tmpfs 256 5 sickbeard /var/opt/sickbeard

couch tmpfs 256 5 couchpotato /home/couchpotato

samba tmpfs 16 5 samba /var/cache/samba ntp tmpfs 8 5 ntp /var/lib/ntp

minidlna tmpfs 16 5 minidlna /var/cache/minidlna

Covered by folder2ram:

log tmpfs 256 8 /var/log vartmp tmpfs 512 5 /var/tmp spool tmpfs 32 5 postfix /var/spool rrd tmpfs 8 5 rrdcached /var/lib/openmediavault/rrd rrdcached tmpfs 256 5 rrdcached /var/lib/rrdcached monit tmpfs 8 5 monit /var/lib/monit php tmpfs 8 5 php5-fpm /var/lib/php5

Example Status: Name Space FileS Real Directory Mounted tmpfs|ramfs ==== ===== ===== ============== =================== rrdcached 16M/256M tmpfs /var/dir2ram/dir2ram_rrdcached /var/lib/rrdcached ntp 1M/8M tmpfs /var/dir2ram/dir2ram_ntp /var/lib/ntp vartmp 0M/512M tmpfs /var/dir2ram/dir2ram_vartmp /var/tmp rrd 1M/8M tmpfs /var/dir2ram/dir2ram_rrd /var/lib/openmediavault/rrd php 1M/8M tmpfs /var/dir2ram/dir2ram_php /var/lib/php5 smart 0M/8M tmpfs /var/dir2ram/dir2ram_smart /var/lib/smartmontools log 21M/256M tmpfs /var/dir2ram/dir2ram_log /var/log monit 1M/8M tmpfs /var/dir2ram/dir2ram_monit /var/lib/monit spool 2M/32M tmpfs /var/dir2ram/dir2ram_spool /var/spool samba 1M/16M tmpfs /var/dir2ram/dir2ram_samba /var/cache/samba

bobafetthotmail commented 8 years ago

oh just browsed fast through it, and it looks cool. you rewrote most parts and it's significantly more pro now.

This weekend will have a more through look into it.

bobafetthotmail commented 8 years ago

gave it a more through look. Is there a compelling reason for it to be written in perl?

I'm not seeing anything that can't be done with shell scripts too, and I'd rather use shell scripts myself.

Reed97123 commented 8 years ago

Hi,

For me shell scripts offer the maximum portability and comparability because it is ensured people can run your shell script without anything else being installed. But their limitation is that they are very limited in their features and what you can do with them.

Programming languages on the other hand are very flexible, but often take compiling or have other installation requirements.

Perl, as a scripting language is a good mix of both benefits. It has near universal support on all systems and is very feature packed.

Off the top of my head I used argument processing, hashes, functions, formatted printing (printf), various regular expression processing, various array functions. I wouldn't have known where to start trying to implement this in a shell script. I'm fairly certain shell scripts don't have an equivalent to perl hash data structures. I also heavily relied upon Perl's regular expression functions which I wouldn't know how or if it is possible to do in a shell script.

I read your GitHub 'Todo List' and I was able to implement nearly everything on your list.

I was curious if there was any other features that should be added from your perspective? I'd be happy to add it in now.

Also, in the time between now and when I posted the version I made a few bug fixes and minor changes. Should I upload the new version to GitHub?

-Reed

On Fri, Jan 22, 2016 at 3:38 AM, bobafetthotmail notifications@github.com wrote:

gave it a more through look. Is there a compelling reason for it to be written in perl?

I'm not seeing anything that can't be done with shell scripts too, and I'd rather use shell scripts myself.

— Reply to this email directly or view it on GitHub https://github.com/bobafetthotmail/folder2ram/pull/1#issuecomment-173893125 .

bobafetthotmail commented 8 years ago

With all due respect because you clearly are a better programmer than me...

I know shell (sh) is limited and I was already thinking about migrating, but I was looking mostly towards bash (which is similar to sh and I understand it decently, it is also the linux default CLI afaik).

Bash can do the things you mentioned as perl advantages.

"I was curious if there was any other features that should be added from your perspective? I'd be happy to add it in now."

The ability to read/parse multiple config files. like most programs do, (like samba) with a /etc/folder2ram/config.d where the lines from the second file override the (same) lines from the first one.

The ability to mount folders to disk, with the synching too. So you can move particular folders inside the RAID volume or data drives/whatever spinning drives. Which will be useful for programs/caches that cannot fit in ram but can't stay on flash either.

Also an option for ram drives to get a frequent synching of ram drives into data drives (either alone or in addition to synching to system drive), like for logs and stuff like that, you can have an option that synches to data drives every X minutes if there is enough write activity in the last X minutes to justify their use, otherwise don't sync and let the data drives go to sleep.

I also wanted to experiment with a script that makes a squashfs of the folder, with a script run at boot that loads and keeps the squashfs in the ram. When it is time to update/upgrade it unmounts the squashfs, runs the update, makes again the squashfs. More for experimenting and seeing if there is any performance increase, this isn't terribly important though.

" Also, in the time between now and when I posted the version I made a few bug fixes and minor changes. Should I upload the new version to GitHub?"

I'm not going to accept Perl scripts in my repo, as I never really used it (yes I'm not kidding, don't look at me like that), and I don't have the time to learn how to use it decently, so I won't be able to maintain what you send.

If you prefer keeping it in Perl, open your own repository, upload it there, maybe mention me in the credits or something, and become the lead developer. You already have a new name for your script lol. I could help with "debianizing" (adding the stuff needed for packaging it for Debian) and changing the OMV plugin to use dir2ram (if you need my help for these at all).

Reed97123 commented 8 years ago

Hi,

I didn't mean to put you on the defensive to defend Bash. I actually don't know much about programming in Bash. I knew enough to follow the logic of your script and understand what you were doing. But it would have been beyond my expertise to add in all the 'todo list' items using Bash. I've used Perl on and off for 16 years, so I was simply more comfortable using it. I already knew how to do 100% of what was needed in Perl whereas I'd probably only know how to do 20% of it in Bash and I'd have to spend a very long time learning.

I did find a very interesting comparison of the two when you mentioned Bash's capabilities. It was written by someone who knows both very well: http://blogs.perl.org/users/buddy_burden/2012/04/perl-vs-shell-scripts.html

I would agree that I should create my own repo. I didn't want to take over your project, but as I'm not good with Bash, I couldn't figure any other way of contributing and accomplishing the needed coding.

Items you mentioned:

For example:

I just created a GitHub repository, with the latest version: https://github.com/Reed97123/dir2ram

I think before considering using it for the flashmemory plugin I'd like to do more testing on it and add in any of the features we think are desired. The config file addition would be super-simple, but might not be needed if the GUI interface is added. I'm also pretty excited about compressed directories in memory, which I think would be a lot of fun to add.

As this is my first GitHub project, any help and advice is definitely welcome!

-Reed

On Sat, Jan 23, 2016 at 12:12 AM, bobafetthotmail notifications@github.com wrote:

With all due respect because you clearly are a better programmer than me...

I know shell (sh) is limited and I was already thinking about migrating, but I was looking mostly towards bash (which is similar to sh and I understand it decently, it is also the linux default CLI afaik).

Bash can do the things you mentioned as perl advantages.

-argument processing http://stackoverflow.com/questions/192249/how-do-i-parse-command-line-arguments-in-bash

-functions :::: I can assure you that there are plenty of functions in my shell script already, go figure in bash. Otherwise I did not understand this point.

-hash tables :::: bash has them http://stackoverflow.com/questions/1494178/how-to-define-hash-tables-in-bash

-formatted printing (printf) ::: there is a "printf" component of coreutils to print formatted text from whatever script language, write a "man printf" from terminal to see what it can do. I'm suspecting it is the same thing called by perl.

  • regular expression processing ::: that's done by calling awk and sed text processors in my own script already, to read the config file for example, or to turn paths into safe names for folders. It's probably easier with perl though, they are VERY through, awk has huge manuals online.

-arrays :::: yes in bash, also with strings http://tldp.org/LDP/abs/html/arrays.html

"I was curious if there was any other features that should be added from your perspective? I'd be happy to add it in now."

The ability to read/parse multiple config files. like most programs do, (like samba) with a /etc/folder2ram/config.d where the lines from the second file override the (same) lines from the first one.

The ability to mount folders to disk, with the synching too. So you can move particular folders inside the RAID volume or data drives/whatever spinning drives. Which will be useful for programs/caches that cannot fit in ram but can't stay on flash either.

I also wanted to experiment with a script that makes a squashfs of the folder, with a script run at boot that loads and keeps the squashfs in the ram. When it is time to update/upgrade it unmounts the squashfs, runs the update, makes again the squashfs. More for experimenting and seeing if there is any performance increase, this isn't terribly important though.

" Also, in the time between now and when I posted the version I made a few bug fixes and minor changes. Should I upload the new version to GitHub?"

I'm not going to accept Perl scripts in my repo, as I never really used it (yes I'm not kidding, don't look at me like that), and I don't have the time to learn how to use it decently, so I won't be able to maintain what you send.

If you prefer keeping it in Perl, open your own repository, upload it there, and maybe mention me in the credits or something, and become the lead developer. You already have a new name for your script lol. I could help with "debianizing" (adding the stuff needed for packaging it for Debian) and changing the OMV plugin to use dir2ram (if you need my help for these at all).

— Reply to this email directly or view it on GitHub https://github.com/bobafetthotmail/folder2ram/pull/1#issuecomment-174160111 .

Reed97123 commented 8 years ago

Hi,

Update about zRam. It is possible to use if you load the backported kernel 3.16. The default OMV Wheezy kernel (3.2) doesn't yet support zRam.

It is actually a great solution, providing a mechanism to have read/write access to a compressed RAM directory. It is a little tedious to add new drives after the initial setup, all drives must be removed and module removed and then readded with the new additional drives. They have added a hot-add features, but it's not in even the backported kernel. Even given the limitation, I think it would make a fantastic addition to the program.

-Reed

On Sat, Jan 23, 2016 at 10:56 AM, Reed Nelson reed.public@gmail.com wrote:

Hi,

I didn't mean to put you on the defensive to defend Bash. I actually don't know much about programming in Bash. I knew enough to follow the logic of your script and understand what you were doing. But it would have been beyond my expertise to add in all the 'todo list' items using Bash. I've used Perl on and off for 16 years, so I was simply more comfortable using it. I already knew how to do 100% of what was needed in Perl whereas I'd probably only know how to do 20% of it in Bash and I'd have to spend a very long time learning.

I did find a very interesting comparison of the two when you mentioned Bash's capabilities. It was written by someone who knows both very well: http://blogs.perl.org/users/buddy_burden/2012/04/perl-vs-shell-scripts.html

I would agree that I should create my own repo. I didn't want to take over your project, but as I'm not good with Bash, I couldn't figure any other way of contributing and accomplishing the needed coding.

Items you mentioned:

  • Multiple configuration files. So this would allow one configuration file from the developer and a second file for the user so they can each change different files and have both loaded so they don't conflict with each others changes?

For example:

  • /etc/dir2ram/dir2ram.conf (Developer controlled file)
  • /etc/dir2ram/dir2ram_user.conf (User controlled file) That was an example off the top of my head. Let me know if you have a better idea. How should it handle duplicates? Currently I ignore redefinitions/duplicates that occur after the first. But I could easily make it so that duplicates overwrite the previous definition. I guess it just depends on what we think would be most useful.
  • Moving directories from one disk to another. So for example you have an OS directory on a flash drive and it is too big for memory, so you want it to effectively be moved to a spin drive/SSD so it doesn't wear out the USB flash drive?
  • Squashfs. I did some reading and I'm curious as to your expected use cases. Squashfs is typically used for live CD Linux distributions and other embedded applications where the read-only aspect isn't a problem. If your desire is to have a compressed directory in memory, perhaps ZRam is a better solution. It would allow us to turn on compression for specified directories. Perhaps similar to this web site: http://askubuntu.com/questions/130374/ramdisk-compressed-writeable-no-swap
  • One idea I had was to allow users to specify directories directly from the OMV GUI interface. This would be similar to Fail2ban's GUI interface. They could enter directories and view a nice summary of what was done there without needing to modify or look at the actual conf file. This might decrease the need for multiple configuration files (earlier todo list item).

I just created a GitHub repository, with the latest version: https://github.com/Reed97123/dir2ram

I think before considering using it for the flashmemory plugin I'd like to do more testing on it and add in any of the features we think are desired. The config file addition would be super-simple, but might not be needed if the GUI interface is added. I'm also pretty excited about compressed directories in memory, which I think would be a lot of fun to add.

As this is my first GitHub project, any help and advice is definitely welcome!

-Reed

On Sat, Jan 23, 2016 at 12:12 AM, bobafetthotmail < notifications@github.com> wrote:

With all due respect because you clearly are a better programmer than me...

I know shell (sh) is limited and I was already thinking about migrating, but I was looking mostly towards bash (which is similar to sh and I understand it decently, it is also the linux default CLI afaik).

Bash can do the things you mentioned as perl advantages.

-argument processing http://stackoverflow.com/questions/192249/how-do-i-parse-command-line-arguments-in-bash

-functions :::: I can assure you that there are plenty of functions in my shell script already, go figure in bash. Otherwise I did not understand this point.

-hash tables :::: bash has them http://stackoverflow.com/questions/1494178/how-to-define-hash-tables-in-bash

-formatted printing (printf) ::: there is a "printf" component of coreutils to print formatted text from whatever script language, write a "man printf" from terminal to see what it can do. I'm suspecting it is the same thing called by perl.

  • regular expression processing ::: that's done by calling awk and sed text processors in my own script already, to read the config file for example, or to turn paths into safe names for folders. It's probably easier with perl though, they are VERY through, awk has huge manuals online.

-arrays :::: yes in bash, also with strings http://tldp.org/LDP/abs/html/arrays.html

"I was curious if there was any other features that should be added from your perspective? I'd be happy to add it in now."

The ability to read/parse multiple config files. like most programs do, (like samba) with a /etc/folder2ram/config.d where the lines from the second file override the (same) lines from the first one.

The ability to mount folders to disk, with the synching too. So you can move particular folders inside the RAID volume or data drives/whatever spinning drives. Which will be useful for programs/caches that cannot fit in ram but can't stay on flash either.

I also wanted to experiment with a script that makes a squashfs of the folder, with a script run at boot that loads and keeps the squashfs in the ram. When it is time to update/upgrade it unmounts the squashfs, runs the update, makes again the squashfs. More for experimenting and seeing if there is any performance increase, this isn't terribly important though.

" Also, in the time between now and when I posted the version I made a few bug fixes and minor changes. Should I upload the new version to GitHub?"

I'm not going to accept Perl scripts in my repo, as I never really used it (yes I'm not kidding, don't look at me like that), and I don't have the time to learn how to use it decently, so I won't be able to maintain what you send.

If you prefer keeping it in Perl, open your own repository, upload it there, and maybe mention me in the credits or something, and become the lead developer. You already have a new name for your script lol. I could help with "debianizing" (adding the stuff needed for packaging it for Debian) and changing the OMV plugin to use dir2ram (if you need my help for these at all).

— Reply to this email directly or view it on GitHub https://github.com/bobafetthotmail/folder2ram/pull/1#issuecomment-174160111 .

bobafetthotmail commented 8 years ago

Closing this without merging for the reasons already discussed.

I've answered (thoroughly) to your post in an issue I opened in your own new repo https://github.com/Reed97123/dir2ram/issues/1