Build faild on Cygwin

[GoogleCodeExporter]

* The attached patch is required for successful build on Cygwin.
* Please, stop setting svn:eol-style to native. It's just disastrous.
* And celebrate the reasonable squid development branch! ;)

$ uname
CYGWIN_NT-6.1-WOW64
$ gcc --version
gcc (GCC) 4.3.4 20090804 (release) 1
Copyright (C) 2008 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Original issue reported on code.google.com by radiant@aol.jp on 18 Mar 2010 at 2:28

Attachments:

• cygwin.diff

[GoogleCodeExporter]

Ok. I'll have to test some of those changes out locally against linux/freebsd - 
I think io.h for example may not 
work. :)

But I'll certainly commit the windows-specific probing stuff. Don't suppose 
you'd be interested in helping me 
migrate some of the platform-specific code out of src/ and into 
platform-specific libraries? I don't like how it's 
sitting in src/ and dotted throughout the rest of the codebase.. :)

Original comment by adrian.c...@gmail.com on 20 Mar 2010 at 8:56

• Changed state: Accepted
• Added labels: Version-Head
• Removed labels: Version-1.0

[GoogleCodeExporter]

Yes, I was too hasty for io.h. Please tweak it to adapt most Unix-alike's.

I'm now working on Windows native and it already builds, works, but not so fine 
for
example COSS and Isssue #91. :)

Code refactoring is certainly needed especially for Windows stuff. But it's too 
heavy
for me and what I want to do is to use Lusca rather to hack, so I keep changes 
small
as far as I can now. (see attach)

IMHO Windows native binary with fancy installer is must for the project's 
success.
Many people want handy caching proxy for Windows but there isn't. I do know 
SquidNT
but it's not so fancy as apache-win...

Original comment by radiant@aol.jp on 20 Mar 2010 at 9:43

Attachments:

• cygwin+mingw.diff

[GoogleCodeExporter]

I've just committed a small part of the work to HEAD. I'll go and evaluate what 
effect those includes have and 
start committing those too.

Hopefully the size of the diff between LUSCA_HEAD and "LUSCA_HEAD that works in 
cygwin" will slowly decrease 
over time. :)

Original comment by adrian.c...@gmail.com on 20 Mar 2010 at 9:59

[GoogleCodeExporter]

Thank you for committing, and the attach is a minimal patch to
build a Windows native binary on cygwin. (cross-compiling targeted to mingw)

Quick note mostly for myself:

  * Cache rebuild doesn't work. (Issue #87, #88, #91)
  * COSS doesn't work yet. (large files)
  * Used memset and strtok for portability.
  * We need better alternative for strtok/strsep. (strtok_r? split?) 
  * More refactoring needed for cleaner build.
  * Get rid of VC++ stuff? It's stale enough...

== Cygwin ==
Requirements:
  * automake
  * gcc4
  * make
  * sharutils (for uudecode)
Build:
  * ./bootstrap.sh
  * ./configure
  * make install

== MinGW (Windows native) ==
Requirements for cross-compiling on Cygwin:
  * gcc-mingw-core
Build:
  * ./bootstrap.sh
  * CC="gcc-3 -mno-cygwin" ./configure --host=i686-pc-mingw32 --enable-win32-service
--prefix=/lusca
  * mkdir /cygdrive/c/lusca
  * ln -s /cygdrive/c/lusca /
  * make install

Original comment by radiant@aol.jp on 22 Mar 2010 at 7:12

Attachments:

• cygwin+mingw.diff

[GoogleCodeExporter]

ok. I'll review the current diff and commit what's needed.

Want to maintain a Wiki page with the requirements and whatnot for getting 
Lusca running on Cygwin?

Original comment by adrian.c...@gmail.com on 22 Mar 2010 at 8:36

[GoogleCodeExporter]

Ok, I'll look at committing some parts of this stuff over the next few days so 
we can minimise the diffs needed.

Thanks again!

Original comment by adrian.c...@gmail.com on 22 Mar 2010 at 11:20

[GoogleCodeExporter]

Ok, I'll take care of the Windows's mess (with a sigh). Give me some bit and 
blame me
if I broke something.

This issue can be closed.

Original comment by radiant@aol.jp on 24 Mar 2010 at 5:09

[GoogleCodeExporter]

Nah, let's keep the issue open for now so we can keep track of progress. :)

Original comment by adrian.c...@gmail.com on 24 Mar 2010 at 5:36

[GoogleCodeExporter]

I mean, Lusca already builds and runs fine on Cygwin, so this one is done. I'll 
file
another issue for MinGW. :)

Original comment by radiant@aol.jp on 24 Mar 2010 at 10:45

[GoogleCodeExporter]

So, memset() instead of bzero() ? Is that something platform specific there?

Original comment by adrian.c...@gmail.com on 24 Mar 2010 at 2:49

[GoogleCodeExporter]

I've been slowly merging in your changes btw. Please update to the latest 
LUSCA_HEAD SVN and regenerate your 
diff!

Original comment by adrian.c...@gmail.com on 24 Mar 2010 at 2:59

[GoogleCodeExporter]

> So, memset() instead of bzero() ? Is that something platform specific there?

Yes,
  * memset() is a C89 standard function but bzero() is a unix's legacy.
  * mingw actually lacks it. (and also err.h and sys/error.h and... too much)

I usually checks by this:
http://www.opengroup.org/onlinepubs/000095399/

Don't worry about another argument on stack; recent compilers are smart enough 
to
expand small memset()s to inline.

And be patient with the wonderful windows doh-full world. :)

Original comment by radiant@aol.jp on 24 Mar 2010 at 9:55

Attachments:

• mingw.patch

[GoogleCodeExporter]

I've migrated bcopy() -> memcpy(). Please regenerate your patch. :)

The next set of changes will be to sanitise the rest of the header includes 
with the
relevant autoconf/automake #ifdefs. I'm going to leave the windows and general
portability related code changes alone until the storage bug fixes have been 
verified.

Original comment by adrian.c...@gmail.com on 25 Mar 2010 at 3:18

• Changed state: Started

[GoogleCodeExporter]

There - I've committed the header file changes to LUSCA_HEAD. Please test and
resubmit an updated patch when you have time.

Original comment by adrian.c...@gmail.com on 25 Mar 2010 at 3:34

[GoogleCodeExporter]

Here it goes..

Still rebuild broken on windows??

2010/03/25 14:15:37| Store rebuilding is  0.0% complete
2010/03/25 14:15:37| WARNING: 1 invalid swap log entries found
2010/03/25 14:15:37| WARNING: 10 invalid swap log entries found
2010/03/25 14:15:37| WARNING: 100 invalid swap log entries found
2010/03/25 14:15:37| WARNING: 1000 invalid swap log entries found
2010/03/25 14:15:36| ufs_rebuild: c:/lusca/var/cache: rebuild type: REBUILD_DISK
2010/03/25 14:15:36| ufs_rebuild: c:/lusca/var/cache: beginning rebuild from 
directory

Original comment by radiant@aol.jp on 25 Mar 2010 at 5:22

Attachments:

• mingw.patch

[GoogleCodeExporter]

Ah, I need to find and commit fixes to the windows IPC stuff too. Hm, let me 
try..

There. Try LUSCA_HEAD now (with your patches, obviously.)

Original comment by adrian.c...@gmail.com on 25 Mar 2010 at 5:32

[GoogleCodeExporter]

I'm happy to be patient. :)

I've committed some more portability fixes and I've begun merging in some of 
the updates to the win32 async IO 
code.

Please drop me another patch when you're ready so I can see what else I have to 
do.

Thanks for your patience with this!

Original comment by adrian.c...@gmail.com on 25 Mar 2010 at 6:18

[GoogleCodeExporter]

hi all today's patch and error on empty cache. :)

2010/03/26 07:11:54| Store rebuilding is  0.0% complete
2010/03/26 07:11:54| WARNING: 1 invalid swap log entries found
2010/03/26 07:11:54| WARNING: 10 invalid swap log entries found
2010/03/26 07:11:54| WARNING: 100 invalid swap log entries found
2010/03/26 07:11:55| WARNING: 1000 invalid swap log entries found
2010/03/26 07:11:54| ufs_rebuild: c:/lusca/var/cache: rebuild type: REBUILD_DISK
2010/03/26 07:11:54| ufs_rebuild: c:/lusca/var/cache: beginning rebuild from 
directory

Original comment by radiant@aol.jp on 25 Mar 2010 at 10:13

Attachments:

• mingw.patch

[GoogleCodeExporter]

Patch revised to build with large files support.

Note that config.h now must be included at the *top* of nearly every source 
files.
This is normal habit in autoconf'ed world so that it shouldn't be so painful for
developpers. :)

Original comment by radiant@aol.jp on 26 Mar 2010 at 1:07

Attachments:

• mingw.patch

[GoogleCodeExporter]

Ok. I'm going to see about committing -just- the rest of the autoconf/automake
changes you've included in that last patch.

inet_ntop() belongs in libsqinet/inet_legacy.c, rather than libcore/tools.c.

The code which includes ../src/squid.h needs to be re-evaluated; nothing 
outside of
src/ should be including anything inside src/.

I really need to move src/win32.c out into a top level library; along with any 
other
OS specific things which obviously shouldn't be in src/.

Anyway. Thanks so far for this work!

Original comment by adrian.c...@gmail.com on 26 Mar 2010 at 7:10

[GoogleCodeExporter]

Yes, yes, ../src/win32.c was too rude, anyway...

A little bit saner version attached.

inet_ntop couldn't be eliminated?

Original comment by radiant@aol.jp on 26 Mar 2010 at 7:36

Attachments:

• mingw.patch

[GoogleCodeExporter]

The autoconf include changes from your patch are now in.

Next - let me look at merging in inet_ntop() from somewhere.

Original comment by adrian.c...@gmail.com on 26 Mar 2010 at 1:12

[GoogleCodeExporter]

Right. I'm going to have to write a configure.in snippet which checks whether
inet_ntop() exists, and if not, compile in a local copy. We do this already for
inet_ntoa() in some specific situations.

Leave it with me. :)

Original comment by adrian.c...@gmail.com on 26 Mar 2010 at 1:42

[GoogleCodeExporter]

Ok, patch revised, and petit announce for windows binary package is ready for 
testing.

http://sites.google.com/site/luscaweb/file-cabinet

Really testing, you know. :)

Original comment by radiant@aol.jp on 30 Mar 2010 at 8:27

Attachments:

• mingw.patch

[GoogleCodeExporter]

Ok. I've committed a couple of little updates to LUSCA_HEAD but right now I 
need to
focus on finding and fixing other rather urgent bugs.

The next thing I think we need to work on is migrating src/win32.c and whatever 
stuff
in squid.h is used by the windows specific code into a separate library, so
libasyncio, libhelper, etc don't end up referencing stuff in src/. That's going 
to
take some time - doubly so since I don't have a Windows install anywhere to 
test with. :)

Original comment by adrian.c...@gmail.com on 31 Mar 2010 at 3:21

[GoogleCodeExporter]

Hi folks I've killed two of ../src/squid.h from my patch and a bug recently
introduced. ;)

Actually, rebuild is still a problem for me. I must be done something wrong... 
or not?

2010/03/31 13:44:19| ufs_rebuild: cache/aufs: rebuild type: REBUILD_LOG
2010/03/31 13:44:19| ufs_rebuild: cache/aufs: rebuild from swaplog
2010/03/31 13:44:19| ufs_rebuild: cache/aufs: 7 of 9 objects read
2010/03/31 13:44:19| WARNING: 1 invalid swap log entries found

Original comment by radiant@aol.jp on 31 Mar 2010 at 4:47

Attachments:

• mingw.patch

[GoogleCodeExporter]

I have to ignore the rebuild errors just for now. It's likely something to do 
with
Windows IPC which I just haven't thought about.

Original comment by adrian.c...@gmail.com on 31 Mar 2010 at 4:56

[GoogleCodeExporter]

Question: in src/win32.c there's these entries for the service definition:

#define VENDOR   "GNU"
#define SOFTWARENAME "Squid"
#define WIN32_VERSION  "2.6"

What about changing it to use PACKAGE_* macros, ie:

#define SOFTWARENAME PACKAGE_TARNAME
#define WIN32_VERSION PACKAGE_VERSION

Would you please try that and let me know if it works out?

Original comment by adrian.c...@gmail.com on 31 Mar 2010 at 5:34

[GoogleCodeExporter]

I've committed more stuff to bring asyncio up to scratch with the changes to 
UNIX.

But UNIX uses pread/pwrite so the seek/read and seek/write is both atomic and 
doens't
interfere with other operations. COSS requires this - it uses the same FD for 
all the
threaded reads and writes which occur.

Is there a readfile/writefile like routine in win32 which takes an file offset +
length for the operation? Ah, use an overlapped structure?

Original comment by adrian.c...@gmail.com on 31 Mar 2010 at 9:14

[GoogleCodeExporter]

>28

It perfectly works but quite useless since I let the installer the job to
install/uninstall the service (i.e. squid.exe itself doesn't create a registry 
key).
It's easier and cleaner so that -i, -r and -n options are no longer needed. (but
needed for compatibility to squid ;-))

>29

Yes, perhaps we need overlapped io

http://msdn.microsoft.com/en-us/library/ms684342%28v=VS.85%29.aspx

or mutex

http://msdn.microsoft.com/en-us/library/aa365541%28v=VS.85%29.aspx

or even transaction api introduced in windows vista. (oh dear...)

http://msdn.microsoft.com/en-gb/magazine/cc163388.aspx

This may be the reason I faild to setup coss storage with SquidNT.

Cygwin's pread/pwrite seems just a seek and read/write...?

Original comment by radiant@aol.jp on 1 Apr 2010 at 12:58

[GoogleCodeExporter]

Ok. So let's just mark COSS as broken for Windows until some underlying storage 
IO
changes are made. The relevant changes are on my TODO list (as I'd like to use 
POSIX
AIO for FreeBSD/Solaris, Event ports for Solaris, etc, etc) but I just don't 
have the
time to do it yet. There's some infrastructure changes needed in the codebase to
efficiently support what's needed for posix aio, overlapped io, etc, etc.

want to do up an updated patch for me? ANd stay in the IRC channel? :P

Original comment by adrian.c...@gmail.com on 1 Apr 2010 at 1:01

[GoogleCodeExporter]

Hours required just to get rid of one ../src/squid.h incompletely.

I heard squid's src is clean long time ago... perhaps decade ago? :)

what? aio? yah, could be...

Original comment by radiant@aol.jp on 1 Apr 2010 at 5:41

Attachments:

• mingw.patch

[GoogleCodeExporter]

Mark as windows specific.

Original comment by adrian.c...@gmail.com on 3 Apr 2010 at 6:08

• Added labels: OpSys-Windows

[GoogleCodeExporter]

Squid? Clean? Haha!

Anyway. I've migrated WIN32_pipe from src/win32.c to lib/win32lib.c . The
declarations are in include/util.h for now.

It's likely I'll create separate files/libraries at some point for various bits 
of
the useful win32 stuff (eg exception handling stuff, thread creation, deletion,
socket handling, etc.)

Original comment by adrian.c...@gmail.com on 3 Apr 2010 at 1:49

[GoogleCodeExporter]

Actually, WIN32_pipe in win32lib.c is problematic. Most fatal is it requires 
some
instances in libiapp to work. (e.g. fd_table and CommStats) So that, we need to 
put
-lmiscutils before -liapp, but -liapp itself requires pipe(). In short, cyclic
dependency. How can I cope with this? That was the reason I put WIN32_pipe to
unnatural place. (I remember it after seeing link error. :-))

../lib/libmiscutil.a(win32lib.o): In function `WIN32_pipe':
workspace/lusca/lib/win32lib.c:817: undefined reference to `_CommStats'
workspace/lusca/lib/win32lib.c:824: undefined reference to `_local_addr'
workspace/lusca/lib/win32lib.c:844: undefined reference to `_local_addr'
workspace/lusca/lib/win32lib.c:842: undefined reference to `_fd_table'
workspace/lusca/lib/win32lib.c:843: undefined reference to `_sqinet_init'
workspace/lusca/lib/win32lib.c:844: undefined reference to 
`_sqinet_set_v4_inaddr'
workspace/lusca/lib/win32lib.c:847: undefined reference to `_fd_table'
workspace/lusca/lib/win32lib.c:848: undefined reference to `_sqinet_init'
workspace/lusca/lib/win32lib.c:849: undefined reference to `_local_addr'
workspace/lusca/lib/win32lib.c:849: undefined reference to 
`_sqinet_set_v4_inaddr'
workspace/lusca/lib/win32lib.c:850: undefined reference to `_local_addr'
collect2: ld returned 1 exit status

Original comment by radiant@aol.jp on 4 Apr 2010 at 12:55

Attachments:

• mingw.patch

[GoogleCodeExporter]

ok. I may create a win32.c file in libiapp/ to put these items for now.

Horrible, but functional.

Original comment by adrian.c...@gmail.com on 4 Apr 2010 at 3:07

[GoogleCodeExporter]

Actually, WIN32_pipe is fine where it is. It shouldn't be doing that Squid-level
socket setup there.

So I'll split WIN32_pipe up into two parts - the "windows" specific stuff and 
the
"squid" level stuff; I'll then stuff the Squid level stuff somewhere else.

The only place where pipe() is called in Lusca is in
libiapp/comm.c:comm_create_fifopair() . That also does the fd_open() calls and 
sets
up the peer related stuff. I'll do something similar for Windows and
conditional-compile it.

Question - libhelper/ipc_win32.c calls _pipe(). What's the difference here? 
Does it
just need to be updated to reflect the _new_ code somehow? or does _pipe() end 
up
calling WIN32_pipe somehow that I can't see? What -uses- WIN32_pipe() (and 
pipe()
calls) ?

Original comment by adrian.c...@gmail.com on 4 Apr 2010 at 5:33

[GoogleCodeExporter]

AFAIK, _pipe() is the windows's native pipe and pipe() is an emulation of 
posix's
pipe (see squid_mswin.h). In Windows, file handles and sockets are different 
things.
Windows's native select() does not work with pipe and that's the reason our 
great
predecessor implemented WIN32_pipe().

http://msdn.microsoft.com/en-us/library/ms740141%28VS.85%29.aspx

Cygwin has an enhanced select() to emulate posix.

http://www.cygwin.com/cygwin-ug-net/highlights.html

Original comment by radiant@aol.jp on 4 Apr 2010 at 6:32

[GoogleCodeExporter]

Ok. God, so much more crap to try and sort through.

There. I migrated it out to libiapp/win32_pipe.[ch] for now. Code which uses 
pipe()
will have to include that header file on windows.

Can you please re-do up a new, up to date patch? I'll look at doing another 
pass or
two merging in your changes to LUSCA_HEAD to try and reduce the diff size 
rather than
increase it. :)

Original comment by adrian.c...@gmail.com on 5 Apr 2010 at 12:17

[GoogleCodeExporter]

Thanks much, but unfortunately WIN32_getrusage also depends on some globals and 
so
temporary moved to win32_pipe.c. d'oh.

Original comment by radiant@aol.jp on 5 Apr 2010 at 11:59

Attachments:

• mingw.patch

[GoogleCodeExporter]

I've merged in some of the changes. I've migrated the _WIN_OS enum from 
src/enums.h
to include/win32_compat.h so it can be used by the routines in lib/win32lib.c 
(ie,
getrusage()).

What else needs to be migrated out of the header files in src/ ?

Also, let's get that Wiki page going so I can at least install cygwin on my 
Windows
XP laptop and compile up Lusca. :)

Original comment by adrian.c...@gmail.com on 6 Apr 2010 at 3:08

[GoogleCodeExporter]

.. FYI, I've got a Windows XP laptop which I'm now installing Cygwin on so I can
actually do compile tests myself. :)

Original comment by adrian.c...@gmail.com on 6 Apr 2010 at 3:17

[GoogleCodeExporter]

ok. The code works fine under Cygwin. I'll next try a mingw build as a service.

Original comment by adrian.c...@gmail.com on 6 Apr 2010 at 11:32

[GoogleCodeExporter]

FYI, recent patches are always (hopefully) available at
http://sites.google.com/site/luscaweb/file-cabinet along with the binary 
packages.
Hint: use 7z to unzip *.msi and inspect docs/ !

Original comment by radiant@aol.jp on 7 Apr 2010 at 12:23

[GoogleCodeExporter]

There's a problem compiling with mingw/nocygwin - io.h being included more than 
once
in some instances.

Any ideas?

Original comment by adrian.c...@gmail.com on 7 Apr 2010 at 12:55

[GoogleCodeExporter]

ok. So there's a -lot- of stuff to try and make Lusca compile natively under 
mingw
rather than using cygwin.

* The place where io.h is included influences whether it works or not. I need to
fiddle some more with this.
* libcore/radix.[ch] gets the windows specific build flags wrong and thus 
doesn't
include the tcpip/winsock headers
* the win32 lib, pipe code is all busted - and now I see why!
* aiops is very busted!
* the win32 IPC helper code is slightly busted

So hm, which version of Cygwin are you using? I'm using the latest (1.7.3) and 
I'm
seeing lots of build issues your patch likely won't fix. :)

Original comment by adrian.c...@gmail.com on 7 Apr 2010 at 4:48

[GoogleCodeExporter]

Strange, I should make the patch to build with only several harmless warnings.

I'm also using the latest plain Cygwin.

I have no idea for now. (so just attach a full build log)

Why??

Header inclusions are certainly strange. If compiled with -mno-cygwin flag,
referenced instances must be under /usr/i686-pc-mingw or 
/usr/lib/gcc/i686-pc-mingw
(I believe so). But the reality is bit different.

gcc-3 -mno-cygwin -dumpspecs

or

echo "#include <io.h>" | gcc-3 -mno-cygwin -E -

may show some hints. (I'm not a gcc guru and can't understand the behavior.)

Original comment by radiant@aol.jp on 7 Apr 2010 at 7:27

Attachments:

• doit.log

[GoogleCodeExporter]

Alrighty. I've committed what seems to be enough code to make the mingw compile 
work,
save for references to strsep and inet_ntop.

Would you mind trying cygwin and mingw compiles; let me know how they go?

Hopefully the level of diffs between LUSCA_HEAD and "works for you" will be a 
lot
smaller now!

Original comment by adrian.c...@gmail.com on 11 Apr 2010 at 3:08

[GoogleCodeExporter]

Hm, a bit more work seems to be needed...

At least ws2tcpip.h must be included in the ipv6 related code. I've mixed it in
squid_mswin.h, but sqinet.h would be more appropriate? Anyway, ws2tcpip.h 
should be
in parallel with netinet/in.h or sys/socket.h .

strsep is not declared in mingw and it is less portable than strtok.

http://www.freebsd.org/cgi/man.cgi?query=strsep

Original comment by radiant@aol.jp on 11 Apr 2010 at 6:08

[GoogleCodeExporter]

right, but strsep is re-entrant; strtok isn't. I'll just import inet_ntop() and
strsep() from freebsd into lib/ somewhere.

I've broken the cygwin build; so now I get to figure out why.

Original comment by adrian.c...@gmail.com on 11 Apr 2010 at 7:42