skoobe / riofs

Userspace S3 filesystem
GNU General Public License v3.0
393 stars 60 forks source link

Is it possible to compile riofs statically? #117

Open davetbo opened 8 years ago

davetbo commented 8 years ago

I've got RioFS working on my development box, but I'd rather not install all the development tools on production boxes. Is it possible to compile RioFS static and then just install the binaries and configuration files on my production servers? Please advise. I checked the ./configure --help options and I didn't see anything obvious. Also, I tried ./configure LDFLAGS="-static" and I get:

[root@dev riofs-master]# ./configure LDFLAGS="-static" checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking target system type... x86_64-unknown-linux-gnu checking for a BSD-compatible install... /bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking whether UID '0' is supported by ustar format... yes checking whether GID '0' is supported by ustar format... yes checking how to create a ustar tar archive... gnutar checking for gcc... gcc checking whether the C compiler works... no configure: error: in /root/riofs-master': configure: error: C compiler cannot create executables Seeconfig.log' for more details

I appreciate your advice and your great code!

wizzard commented 8 years ago

Hello, yes it's possible to statically compile RioFS. The error you got "C compiler cannot create executables" usually means that your system is not configured properly and / or doesn't have installed necessary packages for static compilation.

davetbo commented 8 years ago

I'm running Centos 7 64-bit. Any thoughts on what I might do to resolve the issue? Also, thanks for the super-fast response! :)

On Tue, Jun 28, 2016 at 9:57 AM, Paul Jonkins notifications@github.com wrote:

Hello, yes it's possible to statically compile RioFS. The error you got "C compiler cannot create executables" usually means that your system is not configured properly and / or doesn't have installed necessary packages for static compilation.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/skoobe/riofs/issues/117#issuecomment-229112304, or mute the thread https://github.com/notifications/unsubscribe/AAFCrJaRdoTTpxWfzA--3SbyNwXh0ZBhks5qQVJ0gaJpZM4JATBB .

Best,

Dave

www.ThibaultFineArt.com Painting the Places We Love

wizzard commented 8 years ago

Could you please upload config.log file to any file sharing service and send me the URL so I could take a look at the logs and see what was the problem?

davetbo commented 8 years ago

Here's the config.log https://gist.github.com/davetbo/b825f0acf5fe2e0b4df65177c063bca5 I really appreciate your help!

Also, while I have your attention, would it be an acceptable use case to map multiple web servers to the same bucket using RioFS for their WP uploads folders? If one server changes the bucket (adds or deletes a file), will another server see the changes, or will the caching mechanism cause problems? Are there any non-default configuration options I need to be using for this use case?

Best, Dave

wizzard commented 8 years ago

Ok, the problem is that you don't have "C" compiler installed, please check the "Dependencies" section of Readme file. In order for RioFS more often check for remote updates decrease the value for "dir_cache_max_time" in riofs.conf.xml file.

davetbo commented 8 years ago

I think I do have it installed, and it compiles fine if I don't try to do it as a static job.

Here's everything I can think to look for given your list of dependencies. Each of your dependencies are listed in bold below, with my search results for what's installed following each one. I'm not a C developer, so I'm not sure if there's something else missing.

C Compiler: [root@dev ~]# yum list installed | grep gcc

gcc.x86_64 4.8.5-4.el7 @base

gcc-c++.x86_64 4.8.5-4.el7 @base

gcc-gfortran.x86_64 4.8.5-4.el7 @base

libgcc.x86_64 4.8.5-4.el7 installed

Glib >= 2.22:

[root@dev ~]# yum list installed | grep glib

dbus-glib.x86_64 0.100-7.el7 installed

glib-networking.x86_64 2.42.0-1.el7 installed

glib2.x86_64 2.42.2-5.el7 installed

glib2-devel.x86_64 2.42.2-5.el7 @base

glibc.x86_64 2.17-106.el7_2.6 @updates

glibc-common.x86_64 2.17-106.el7_2.6 @updates

glibc-devel.x86_64 2.17-106.el7_2.6 @updates

glibc-headers.x86_64 2.17-106.el7_2.6 @updates

fuse >= 2.7.3

[root@dev ~]# yum list installed | grep fuse

fuse.x86_64 2.9.2-6.el7 @base

fuse-devel.x86_64 2.9.2-6.el7 @base

fuse-libs.x86_64 2.9.2-6.el7 @base

libevent >= 2.0

[root@dev ~]# yum list installed | grep libevent

libevent.x86_64 2.0.21-4.el7 installed

libevent-devel.x86_64 2.0.21-4.el7 @base

libxml >= 2.6

[root@dev ~]# yum list installed | grep libxml

libxml2.x86_64 2.9.1-6.el7_2.3 @updates

libxml2-devel.x86_64 2.9.1-6.el7_2.3 @updates

libcrypto >= 0.9

[root@dev ~]# yum list installed | grep openssl

openssl.x86_64 1:1.0.1e-51.el7_2.5 @updates

openssl-devel.x86_64 1:1.0.1e-51.el7_2.5 @updates

openssl-libs.x86_64 1:1.0.1e-51.el7_2.5 @updates

libmagic (I had to guess on this one, I think I found it and it looks installed, but I'm not calling it on the ./configure anyway)

[root@dev ~]# locate libmagic

/usr/lib64/libmagic.so.1

/usr/lib64/libmagic.so.1.0.0

/usr/share/doc/php56w-common-5.6.22/libmagic_LICENSE

[root@dev ~]# yum provides libmagic.so.1

file-libs-5.11-31.el7.i686 : Libraries for applications using libmagic

Repo : base

Matched from:

Provides : libmagic.so.1

[root@dev ~]# yum list installed | grep file-libs

file-libs.x86_64 5.11-31.el7 installed

Is there something other than that? Was there a line in particular in the config.log that made it look like I didn't have the C compiler installed? Which specific options should I add to ./configure to try to compile it as static? Is LDFLAGS="-static" attempting to do it as 32-bit or something?

Best, Dave

On Tue, Jun 28, 2016 at 12:47 PM, Paul Jonkins notifications@github.com wrote:

Ok, the problem is that you don't have "C" compiler installed, please check the "Dependencies" section of Readme file. In order for RioFS more often check for remote updates decrease the value for "dir_cache_max_time" in riofs.conf.xml file.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/skoobe/riofs/issues/117#issuecomment-229161131, or mute the thread https://github.com/notifications/unsubscribe/AAFCrN7TEbGn1Wt6a3TGw-cbf3IKYE5Qks5qQXo-gaJpZM4JATBB .

Best,

Dave

www.ThibaultFineArt.com Painting the Places We Love

davetbo commented 8 years ago

I followed these installation instructions: https://www.linkedin.com/pulse/mount-s3-bucket-linux-using-riofs-alternative-s3fs-saranjeet-singh?forceNoSplash=true

They looked like yours here: https://github.com/skoobe/riofs/wiki/Centos

But yours appear to cut off, whereas these kept going.

I believe yum groupinstall "Development Tools" should have done the trick, but I could be wrong.

Best, Dave

On Tue, Jun 28, 2016 at 1:19 PM, David Thibault dave@thibaultfineart.com wrote:

I think I do have it installed, and it compiles fine if I don't try to do it as a static job.

Here's everything I can think to look for given your list of dependencies. Each of your dependencies are listed in bold below, with my search results for what's installed following each one. I'm not a C developer, so I'm not sure if there's something else missing.

C Compiler: [root@dev ~]# yum list installed | grep gcc

gcc.x86_64 4.8.5-4.el7 @base

gcc-c++.x86_64 4.8.5-4.el7 @base

gcc-gfortran.x86_64 4.8.5-4.el7 @base

libgcc.x86_64 4.8.5-4.el7 installed

Glib >= 2.22:

[root@dev ~]# yum list installed | grep glib

dbus-glib.x86_64 0.100-7.el7 installed

glib-networking.x86_64 2.42.0-1.el7 installed

glib2.x86_64 2.42.2-5.el7 installed

glib2-devel.x86_64 2.42.2-5.el7 @base

glibc.x86_64 2.17-106.el7_2.6 @updates

glibc-common.x86_64 2.17-106.el7_2.6 @updates

glibc-devel.x86_64 2.17-106.el7_2.6 @updates

glibc-headers.x86_64 2.17-106.el7_2.6 @updates

fuse >= 2.7.3

[root@dev ~]# yum list installed | grep fuse

fuse.x86_64 2.9.2-6.el7 @base

fuse-devel.x86_64 2.9.2-6.el7 @base

fuse-libs.x86_64 2.9.2-6.el7 @base

libevent >= 2.0

[root@dev ~]# yum list installed | grep libevent

libevent.x86_64 2.0.21-4.el7 installed

libevent-devel.x86_64 2.0.21-4.el7 @base

libxml >= 2.6

[root@dev ~]# yum list installed | grep libxml

libxml2.x86_64 2.9.1-6.el7_2.3 @updates

libxml2-devel.x86_64 2.9.1-6.el7_2.3 @updates

libcrypto >= 0.9

[root@dev ~]# yum list installed | grep openssl

openssl.x86_64 1:1.0.1e-51.el7_2.5 @updates

openssl-devel.x86_64 1:1.0.1e-51.el7_2.5 @updates

openssl-libs.x86_64 1:1.0.1e-51.el7_2.5 @updates

libmagic (I had to guess on this one, I think I found it and it looks installed, but I'm not calling it on the ./configure anyway)

[root@dev ~]# locate libmagic

/usr/lib64/libmagic.so.1

/usr/lib64/libmagic.so.1.0.0

/usr/share/doc/php56w-common-5.6.22/libmagic_LICENSE

[root@dev ~]# yum provides libmagic.so.1

file-libs-5.11-31.el7.i686 : Libraries for applications using libmagic

Repo : base

Matched from:

Provides : libmagic.so.1

[root@dev ~]# yum list installed | grep file-libs

file-libs.x86_64 5.11-31.el7 installed

Is there something other than that? Was there a line in particular in the config.log that made it look like I didn't have the C compiler installed? Which specific options should I add to ./configure to try to compile it as static? Is LDFLAGS="-static" attempting to do it as 32-bit or something?

Best, Dave

On Tue, Jun 28, 2016 at 12:47 PM, Paul Jonkins notifications@github.com wrote:

Ok, the problem is that you don't have "C" compiler installed, please check the "Dependencies" section of Readme file. In order for RioFS more often check for remote updates decrease the value for "dir_cache_max_time" in riofs.conf.xml file.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/skoobe/riofs/issues/117#issuecomment-229161131, or mute the thread https://github.com/notifications/unsubscribe/AAFCrN7TEbGn1Wt6a3TGw-cbf3IKYE5Qks5qQXo-gaJpZM4JATBB .

Best,

Dave

www.ThibaultFineArt.com Painting the Places We Love

Best,

Dave

www.ThibaultFineArt.com Painting the Places We Love

wizzard commented 8 years ago

could you please create a small C file (called test.c):

int main() {
return 0;
}

and try to compile it: gcc -static test.c

davetbo commented 8 years ago

I got this error:

/bin/ld: cannot find -lc

collect2: error: ld returned 1 exit status

I googled bin ld cannot find lc, and found an article mentioning that I needed glibc-static installed. I installed it, and the test file compiled fine.

Along those same lines, I also found and installed openssl-static and libxml2-static:

[root@dev riofs-master]# yum list installed | grep static

glibc-static.x86_64 2.17-106.el7_2.6 @updates

libxml2-static.x86_64 2.9.1-6.el7_2.3 @updates

openssl-static.x86_64 1:1.0.1e-51.el7_2.5 @updates

Now when I do ./configure LDFLAGS="-static" I get:

configure: error: could not find OpenSSL

Here's the new config.log:

https://gist.github.com/davetbo/91b6a66d0ddb80c9b8fcc3874e4f0225

Thanks for all your help!

Best,

Dave

On Tue, Jun 28, 2016 at 1:33 PM, Paul Jonkins notifications@github.com wrote:

could you please create a small C file (called test.c):

int main() { return 0; }

and try to compile it: gcc -static test.c

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/skoobe/riofs/issues/117#issuecomment-229174120, or mute the thread https://github.com/notifications/unsubscribe/AAFCrKYHoTvtHNRsIl4GhTtlU6GBrn2fks5qQYUVgaJpZM4JATBB .

Best,

Dave

www.ThibaultFineArt.com Painting the Places We Love

davetbo commented 8 years ago

OK, until we can figure out the static compiles I'm using it the dynamically linked way on my dev box to test it out. I just used a script to cut up a high res image into a bunch of small res images. I've used it before and it works in general (I was using the same script on yas3fs and it was working, but it was really slow).

Anyway, I'm getting a really weird problem. When I go to the output folder, I get half of the files showing up as "no such file or directory" when I do an ls. See the screenshot here:

https://drive.google.com/file/d/0B5sV9QxXFdWsTWViNDlUWkkxdmM/view?usp=sharing

The black & red ones are showing up as non-existent (but still showing there for some reason) and the purple/pink ones are working as expected. Any idea what could cause this? I'm pretty sure it's specific to riofs, since this same script was working before and works fine on a local directory.

Best, Dave

On Tue, Jun 28, 2016 at 1:57 PM, David Thibault dave@thibaultfineart.com wrote:

I got this error:

/bin/ld: cannot find -lc

collect2: error: ld returned 1 exit status

I googled bin ld cannot find lc, and found an article mentioning that I needed glibc-static installed. I installed it, and the test file compiled fine.

Along those same lines, I also found and installed openssl-static and libxml2-static:

[root@dev riofs-master]# yum list installed | grep static

glibc-static.x86_64 2.17-106.el7_2.6 @updates

libxml2-static.x86_64 2.9.1-6.el7_2.3 @updates

openssl-static.x86_64 1:1.0.1e-51.el7_2.5 @updates

Now when I do ./configure LDFLAGS="-static" I get:

configure: error: could not find OpenSSL

Here's the new config.log:

https://gist.github.com/davetbo/91b6a66d0ddb80c9b8fcc3874e4f0225

Thanks for all your help!

Best,

Dave

On Tue, Jun 28, 2016 at 1:33 PM, Paul Jonkins notifications@github.com wrote:

could you please create a small C file (called test.c):

int main() { return 0; }

and try to compile it: gcc -static test.c

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/skoobe/riofs/issues/117#issuecomment-229174120, or mute the thread https://github.com/notifications/unsubscribe/AAFCrKYHoTvtHNRsIl4GhTtlU6GBrn2fks5qQYUVgaJpZM4JATBB .

Best,

Dave

www.ThibaultFineArt.com Painting the Places We Love

Best,

Dave

www.ThibaultFineArt.com Painting the Places We Love

wizzard commented 8 years ago

Regarding the static build issues: unfortunately I don't have an access to an installed CentOS computer to test it, but it seems that "-lssl" is missing. Could you try to prepend LDLIBS = -lssl in front of ./configure script ?

Regarding "no such file or directory" issue: have you modified the default riofs.conf.xml file? Could you tell me how many files are there in the directory? Possible issue is that you trying to get the list of files before the directory is fully filled with the files.

davetbo commented 8 years ago

Paul,

Sorry to let this drop a few days. I tried this command:

./configure LDFLAGS="-static" LDLIBS="-lssl"

and it resulted in the same result. I also tried

./configure LDFLAGS="-static -L/usr/lib64" --libdir=/usr/lib64 LDLIBS="-lssl -lcrypto" (based on some other articles I found) and got the same result still. See this gist:

https://gist.github.com/davetbo/a85adcec6e5b0e0a5f2566583452345a

Best, Dave

On Wed, Jun 29, 2016 at 7:26 AM, Paul Jonkins notifications@github.com wrote:

Regarding the static build issues: unfortunately I don't have an access to an installed CentOS computer to test it, but it seems that "-lssl" is missing. Could you try to prepend LDLIBS = -lssl in front of ./configure script ?

Regarding "no such file or directory" issue: have you modified the default riofs.conf.xml file? Could you tell me how many files are there in the directory? Possible issue is that you trying to get the list of files before the directory is fully filled with the files.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/skoobe/riofs/issues/117#issuecomment-229372717, or mute the thread https://github.com/notifications/unsubscribe/AAFCrDhjG5r6yMmml1XInyZvUHN75SDMks5qQoCngaJpZM4JATBB .

Best,

Dave

www.ThibaultFineArt.com Painting the Places We Love