search

jablko / dedup

Try not to download the same file twice. Improve cache efficiency and speed up downloads.
https://cwiki.apache.org/confluence/display/TS/Metalink
6 stars 1 forks source link

ATS crashes when Metalink Enabled in plugins.config #1

Open Degreane opened 10 years ago

Degreane commented 10 years ago

Hi I downloaded several versions of ATS that are available (4.1.x ~ 4.2.x ) and all compiled with --enable-experimental-plugins as described in the manual ./configure --help shows

nevertheless whenever i enable the plugin metalink traffic_server crashes and the manager keeps restarting it

tried it with different versions of linux. (ubuntu 13.10 - fedora 18,19,20) same issue

Much Regards

[Jan 27 18:47:51.693] Manager {0xb7609780} NOTE: [ClusterCom::ClusterCom] Node running on OS: 'Linux' Release: '3.12.8-300.fc20.i686+PAE' [Jan 27 18:47:51.693] Manager {0xb7609780} NOTE: [LocalManager::listenForProxy] Listening on port: 8080 [Jan 27 18:47:51.693] Manager {0xb7609780} NOTE: [TrafficManager] Setup complete [Jan 27 18:47:52.712] Manager {0xb7609780} NOTE: [LocalManager::startProxy] Launching ts process [Jan 27 18:47:52.717] Manager {0xb7609780} NOTE: [LocalManager::pollMgmtProcessServer] New process connecting fd '13' [Jan 27 18:47:52.717] Manager {0xb7609780} NOTE: [Alarms::signalAlarm] Server Process born

[Jan 27 18:48:35.991] Manager {0xb7609780} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104) [Jan 27 18:48:35.991] Manager {0xb7609780} NOTE: [LocalManager::mgmtShutdown] Executing shutdown request. [Jan 27 18:48:35.991] Manager {0xb7609780} NOTE: [LocalManager::processShutdown] Executing process shutdown request. [Jan 27 18:48:35.991] Manager {0xb7609780} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message [Jan 27 18:48:35.991] Manager {0xb7609780} ERROR: (last system error 32: Broken pipe)

hopefully you may check it out and get it fixed ..

antbryan commented 10 years ago

from @jablko

I pushed the updated plugin to GitHub (hopefully it will also be distributed with the next Traffic Server release). To build it,

1) download the updated metalink.cc file 2) replace the plugins/experimental/metalink/metalink.cc file in your Traffic Server source tree with the updated file, 3) and rebuild Traffic Server by rerunning "make".

Degreane commented 10 years ago

excuse me but when did you update it ???

much regards

     Faysal Banna

Meteorological Services Rafic Harriri International Airport Beirut - Lebanon Mob: +961-3-258043

On 02/07/2014 10:50 PM, antbryan wrote:

from @jablko https://github.com/jablko

I pushed the updated plugin to GitHub (hopefully it will also be distributed with the next Traffic Server release). To build it,

1) download the updated metalink.cc file 2) replace the plugins/experimental/metalink/metalink.cc file in your Traffic Server source tree with the updated file, 3) and rebuild Traffic Server by rerunning "make".

— Reply to this email directly or view it on GitHub https://github.com/jablko/dedup/issues/1#issuecomment-34501346.

antbryan commented 10 years ago

if you look at the commit log, you can see that @jablko updated it a few days ago on 4 & 5 Feb.

have you tried the new code since then?

it's also been included upstream, so hopefully should be available w/ the next ATS release: https://git-wip-us.apache.org/repos/asf?p=trafficserver.git

Degreane commented 10 years ago

I already used this metalink.cc with ATS 5.0.x and 4.1.2 and both give this error below when i activate the metalink plugin...

did you compile it and test it yourself ?

anyhow the error is as shows [TrafficServer] using root directory '/usr/local' FATAL: InkAPI.cc:6248: failed assert sdk_sanity_check_iocore_structure(connp) == TS_SUCCESS /usr/local/bin/traffic_server - STACK TRACE: /usr/local/lib/libtsutil.so.4(+0xf36b)[0x4001136b] /usr/local/lib/libtsutil.so.4(+0xe23d)[0x4001023d] /usr/local/bin/traffic_server(TSVConnWrite+0x2c9)[0x80f4489] /usr/local/libexec/trafficserver/metalink.so(+0x1d0f)[0x40047d0f] /usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x15f)[0x83087bf] /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x7a3)[0x83093c3] /usr/local/bin/traffic_server(main+0xc6a)[0x80bc04a] /lib/libc.so.6(__libc_start_main+0xf3)[0x47794b73] /usr/local/bin/traffic_server[0x80c134d] [TrafficServer] using root directory '/usr/local' FATAL: InkAPI.cc:6248: failed assert sdk_sanity_check_iocore_structure(connp) == TS_SUCCESS /usr/local/bin/traffic_serverFATAL: InkAPI.cc:6248: failed assert sdk_sanity_check_iocore_structure(connp) == TS_SUCCESS

Degreane commented 10 years ago

try to compile it yourself and when you get it running and tested please post your procedure here so we may get it running ....

tested with ATS 4.1.2 and 5.0.x and still same error

what debug procedure you wish me to do so we can ease up the issue of fixing it up ?

kieronrob commented 10 years ago

Hi,

I get these errors when the plugin is enabled on ATS 4.2.1:

FATAL: InkAPI.cc:6261: failed assert sdk_sanity_check_iocore_structure(connp) == TS_SUCCESS /usr/local/bin/traffic_server - STACK TRACE: /usr/local/lib/libtsutil.so.4(+0x1f5d8)[0x2b82d6c3c5d8] /usr/local/lib/libtsutil.so.4(+0x1da3f)[0x2b82d6c3aa3f] /usr/local/bin/traffic_server(TSVConnWrite+0x11f)[0x4b507f] /usr/local/libexec/trafficserver/metalink.so(+0x1ffc)[0x2b82de5f4ffc] /usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x6ab7cf] /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x61b)[0x6ac2fb] /usr/local/bin/traffic_server(main+0xe1b)[0x4c511b] /lib64/libc.so.6(__libc_start_main+0xfd)[0x3d2921ed1d] /usr/local/bin/traffic_server[0x487e69]

Please let me know if you need any additional info.

jablko commented 10 years ago

Hi, are you able to recompile Traffic Server? If so, can you please try the following,

1) download the latest metalink.cc code [1] 2) replace the plugins/experimental/metalink/metalink.cc in your Traffic Server source tree with the latest metalink.cc code 3) and recompile Traffic Server by running "make"

[1] https://raw.github.com/jablko/dedup/master/metalink.cc

After recompiling and restarting Traffic Server, do you still get these errors?

kieronrob commented 10 years ago

Hi Jack,

Recompiled ATS with the updated metalink.cc and the erors are no longer coming up.

Thank you for the swift reply and resolution as this is a feature I really think is valuable.

Regards,

Kieron

jablko commented 10 years ago

Great! If you encounter any more problems, please let me know. The latest metalink.cc code should be included in the next major Traffic Server release (5.X)

kieronrob commented 10 years ago

Hi Jack,

I have setup a new box with ATS and included the new metalink.cc in the /plugins/experimental/metalink directory before running ./configure --enable--experimental-plugins.

I am still getting the same error:

FATAL: InkAPI.cc:4534: failed assert sdk_sanity_check_txn(txnp) == TS_SUCCESS /usr/local/bin/traffic_server - STACK TRACE: /usr/local/lib/libtsutil.so.4(+0x1a9d4)[0x4001c9d4] /usr/local/lib/libtsutil.so.4(+0x191dd)[0x4001b1dd] /usr/local/bin/traffic_server(TSHttpTxnClientReqGet+0x32)[0x80df972] /usr/local/libexec/trafficserver/metalink.so(+0x18d6)[0x400348d6] /usr/local/bin/traffic_server(_ZN7CacheVC8callcontEi+0x5a)[0x821ac9a] /usr/local/bin/traffic_server(_ZN7CacheVC19openWriteStartBeginEiP5Event+0x288)[0x8297b38] /usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8c)[0x82dbd9c] /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x5ec)[0x82dc87c] /usr/local/bin/traffic_server[0x82db17c] /lib/libpthread.so.0[0xbd1b39] /lib/libc.so.6(clone+0x5e)[0xb0dd8e]

Are there any other things I can try as I would like to make use of the plugin.

I am also using the cacheurl plugin and can send you a copy of my records.config if it will help.

Kind regards,

Kieron

On Sun, May 11, 2014 at 11:12 PM, Jack Bates notifications@github.comwrote:

Great! If you encounter any more problems, please let me know. The latest metalink.cc code should be included in the next major Traffic Server release (5.X)

— Reply to this email directly or view it on GitHubhttps://github.com/jablko/dedup/issues/1#issuecomment-42783451 .

Kieron Robertson

Man and Machine

Mail: kieron@manandmachine.biz Cell: 083 309 3538

jablko commented 10 years ago

Hmm, first I'm curious if this crash is specific to the version of Traffic Server or not. If you can recompile Traffic Server, can you please reproduce the crash with the latest code from the Traffic Server repository?

The following commands will check out and build the latest code from the repository:

$ git clone https://git-wip-us.apache.org/repos/asf/trafficserver.git $ cd trafficserver $ autoreconf -i $ ./configure --enable-experimental-plugins $ make

You don't need to replace the metalink.cc file in this case because the repository already contains the latest metalink.cc code.

Next, can you please run Traffic Server with the "http" debug tag and send me the output when it crashes? Run traffic_server with the "-Thttp" switch or run traffic_manager with the "-debug http" option.

Thanks for reporting this crash. With the extra details above, I will work on fixing it. If you have any questions or problems, please let me know.

kieronrob commented 10 years ago

Hi Jack,

Sorry for the long delay, but been a bit hectic. I have recompiled and reinstalled Traffic Server as per your suggestions above and am now running it with the trafficserver start -Thttp switch.

I will come back to you if it crashes and give you the logs.

Regards,

Kieron

On Fri, May 23, 2014 at 6:52 PM, Jack Bates notifications@github.com wrote:

Hmm, first I'm curious if this crash is specific to the version of Traffic Server or not. If you can recompile Traffic Server, can you please reproduce the crash with the latest code from the Traffic Server repository?

The following commands will check out and build the latest code from the repository:

$ git clone https://git-wip-us.apache.org/repos/asf/trafficserver.git $ cd trafficserver $ autoreconf -i $ ./configure --enable-experimental-plugins $ make

You don't need to replace the metalink.cc file in this case because the repository already contains the latest metalink.cc code.

Next, can you please run Traffic Server with the "http" debug tag and send me the output when it crashes? Run traffic_server with the "-Thttp" switch or run traffic_manager with the "-debug http" option.

Thanks for reporting this crash. With the extra details above, I will work on fixing it. If you have any questions or problems, please let me know.

— Reply to this email directly or view it on GitHub https://github.com/jablko/dedup/issues/1#issuecomment-44034969.

Kieron Robertson

Man and Machine

Mail: kieron@manandmachine.biz Cell: 083 309 3538

kieronrob commented 10 years ago

Hi,

OK - got a crash. Here is the output on the CLI:

[root@ats ~]# traffic_manager -debug http [E. Mgmt] log ==> [TrafficManager] using root directory '/usr/local' [Jun 3 10:14:50.803] Manager {0x7f4b7a9cb7e0} DEBUG: (http_tproxy) Listen port 8080 inbound transparency enabled. [TrafficServer] using root directory '/usr/local' FATAL: InkAPI.cc:4535: failed assert sdk_sanity_check_txn(txnp) == TS_SUCCESS /usr/local/bin/traffic_server - STACK TRACE: /usr/local/lib/libtsutil.so.5(+0x1f5b8)[0x2b486fd7e5b8] /usr/local/lib/libtsutil.so.5(+0x1da7f)[0x2b486fd7ca7f] /usr/local/bin/traffic_server(TSHttpTxnClientReqGet+0x26)[0x4b6526] /usr/local/libexec/trafficserver/metalink.so(+0x1fd2)[0x2b487c203fd2] /usr/local/bin/traffic_server(_ZN7CacheVC8callcontEi+0x53)[0x669583] /usr/local/bin/traffic_server(_ZN7CacheVC19openWriteStartBeginEiP5Event+0x2d8)[0x6e2f08] /usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x7296bf] /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x663)[0x72a233] /usr/local/bin/traffic_server[0x728a6a] /lib64/libpthread.so.0(+0x38a88079d1)[0x2b48705079d1] /lib64/libc.so.6(clone+0x6d)[0x38a84e8b6d]

And the second SSH session I had open:

[connect] ERROR (main_socket_fd 3): Connection refused traffic_top: stats.h:251: void Stats::getStats(): Assertion `TSRecordGetInt(item.name, &value) == TS_ERR_OKAY' failed. Aborted (core dumped) [root@ats ~]#

Manager Log:

[Jun 3 11:02:16.279] Manager {0x7f4b7a9cb7e0} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104) [Jun 3 11:02:16.279] Manager {0x7f4b7a9cb7e0} NOTE: [LocalManager::mgmtShutdown] Executing shutdown request. [Jun 3 11:02:16.279] Manager {0x7f4b7a9cb7e0} NOTE: [LocalManager::processShutdown] Executing process shutdown request. [Jun 3 11:02:16.280] Manager {0x7f4b7a9cb7e0} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message [Jun 3 11:02:16.280] Manager {0x7f4b7a9cb7e0} ERROR: (last system error 32: Broken pipe)

Please let me know if you need anything else.

Regards,

Kieron

On Tue, Jun 3, 2014 at 10:12 AM, Kieron Robertson kieron@manandmachine.biz wrote:

Hi Jack,

Sorry for the long delay, but been a bit hectic. I have recompiled and reinstalled Traffic Server as per your suggestions above and am now running it with the trafficserver start -Thttp switch.

I will come back to you if it crashes and give you the logs.

Regards,

Kieron

On Fri, May 23, 2014 at 6:52 PM, Jack Bates notifications@github.com wrote:

Hmm, first I'm curious if this crash is specific to the version of Traffic Server or not. If you can recompile Traffic Server, can you please reproduce the crash with the latest code from the Traffic Server repository?

The following commands will check out and build the latest code from the repository:

$ git clone https://git-wip-us.apache.org/repos/asf/trafficserver.git $ cd trafficserver $ autoreconf -i $ ./configure --enable-experimental-plugins $ make

You don't need to replace the metalink.cc file in this case because the repository already contains the latest metalink.cc code.

Next, can you please run Traffic Server with the "http" debug tag and send me the output when it crashes? Run traffic_server with the "-Thttp" switch or run traffic_manager with the "-debug http" option.

Thanks for reporting this crash. With the extra details above, I will work on fixing it. If you have any questions or problems, please let me know.

— Reply to this email directly or view it on GitHub https://github.com/jablko/dedup/issues/1#issuecomment-44034969.

Kieron Robertson

Man and Machine

Mail: kieron@manandmachine.biz Cell: 083 309 3538

Kieron Robertson

Man and Machine

Mail: kieron@manandmachine.biz Cell: 083 309 3538

jablko commented 10 years ago

No sweat, thanks for reproducing the crash with the latest Traffic Server code. I was wrong about how to turn on the "http" debug tag, sorry! Can you please run either "trafficserver start" or traffic_manager with the following two environment variables and send me the output again?

PROXY_CONFIG_DIAGS_DEBUG_ENABLED=1 PROXY_CONFIG_DIAGS_DEBUG_TAGS=http

Thanks for your help tracking down this crash. If you have any questions or problems, please let me know.

kieronrob commented 10 years ago

Hi Jack,

Just an update - I haven't had another crash since the last one, so looking good for the re-compiled ats from source.

I will use this method on another test appliance and give you feedback if I see the crash again.

Kind regards,

Kieron

On Tue, Jun 3, 2014 at 10:21 PM, Jack Bates notifications@github.com wrote:

No sweat, thanks for reproducing the crash with the latest Traffic Server code. I was wrong about how to turn on the "http" debug tag, sorry! Can you please run either "trafficserver start" or traffic_manager with the following two environment variables and send me the output again?

PROXY_CONFIG_DIAGS_DEBUG_ENABLED=1 PROXY_CONFIG_DIAGS_DEBUG_TAGS=http

Thanks for your help tracking down this crash. If you have any questions or problems, please let me know.

— Reply to this email directly or view it on GitHub https://github.com/jablko/dedup/issues/1#issuecomment-45015153.

Kieron Robertson

Man and Machine

Mail: kieron@manandmachine.biz Cell: 083 309 3538

jablko commented 10 years ago

Thanks for keeping me updated, do let me know if you experience any more crashes. I'll work on fixing them if you do.

yzlai commented 10 years ago

I got same crash report. It happened while calling TSHttpTxnClientReqGet() from cache_open_write() function.

-Ethan