perfsonar / project

The perfSONAR project's primary wiki and issue tracker.
Apache License 2.0
53 stars 10 forks source link

rebuild ixgbe driver for future pSPT release #694

Closed arlake228 closed 9 years ago

arlake228 commented 9 years ago

Original issue 695 created by arlake228 on 2013-02-19T19:01:34.000Z:

Communication with end site follows. They saw issues with ixgbe driver, and DYNES team rebuilt for their use. Future pSPT releases should do the same:

-=-=-=-=-=-=-=-=-=-=-=-=-=-

My recollection is that ixgbe just requires a rebuild with the new kernel. It should just be a matter of rebuilding the most recent srpm when we rebuild the kernels.

Cheers, Aaron

On 2/19/13 7:35 AM, "Jason Zurawski" <zurawski@internet2.edu> wrote:

Gang;

See attached message regarding some testing (via the DYNES effort) of the pSPT on Dell R610 hardware that requires the ixgbe driver for the 10G NIC. Since this may come back to bite everyone regarding support on this hardware, would it make sense to take some of the work that Ben provided below, and get that rolled into a future RC?

Thanks;

-jason

Begin forwarded message:

From: Benjeman Meekhof <bmeekhof@umich.edu> Subject: Re: 10G driver problem on FRGP perfSONAR machine Date: February 18, 2013 1:42:00 PM EST To: Pete Siemsen <siemsen@ucar.edu> Cc: Jason Zurawski <zurawski@internet2.edu>, Shawn McKee <smckee@umich.edu>, Brian Cashman <bsc@internet2.edu>, John Hernandez <jph@ucar.edu>

I'm sorry, I misread the SPEC file. You do not need to do any manual copying of .ko files. The rpm install process does that. We do still need to build an RPM appropriate to the kernel on these machines.

-Ben

On Mon, Feb 18, 2013 at 1:29 PM, Benjeman Meekhof <bmeekhof@umich.edu> wrote: Hi Pete,

I apologize in advance that this is more steps than I would like. I may be able to take care of some for you with more information but we'll start here.

I've built an ixgbe source RPM (SRPM) which incorporates a patch to enable the driver to use any brand of SFP module. I've tested it can build on an el5 type system. Without knowing what enterprise version and what kernel we are dealing with, you can do the following to create a binary RPM for your system. You or someone else here can tell me those variables and I'll see if I can build one specific to your perfSonar system to save steps 1-4 for you.

  1. Download the SRPM: http://dynes.grid.umich.edu/dynes/repo/SRPMS/ixgbe-3.12.6-1SFP.src.rpm (let me know if you need to get there from a network besides your DYNES system network)
  2. rpm -ivh ixgbe-3.12.6-1SFP.src.rpm
  3. (assuming EL5-based) rpmbuild -bb /usr/src/redhat/SPECS/ixgbe.spec
  4. rpm -ivh /usr/src/redhat/RPMS/x86_64/ixgbe-3.12.6-1SFP.x86_64.rpm
  5. Since the kernel packages ixgbe.ko our package would conflict if we named it ixgbe.ko. We have to manually over-write the old one (this path is an example, it is /lib/modules/<your running kernel>: mv /lib/modules/2.6.18-308.16.1.el5/kernel/drivers/net/ixgbe/ixgbe.ko /lib/modules/2.6.18-308.16.1.el5/kernel/drivers/net/ixgbe/ixgbe.ko.old mv /lib/modules/2.6.18-308.16.1.el5/kernel/drivers/net/ixgbe/ixgbe.ko.new /lib/modules/2.6.18308.16.1.el5/kernel/drivers/net/ixgbe/ixgbe.ko

Sorry it's such a pain. We might actually be better off doing what I didn't want to do and building a special perfSonar kernel based on the stock one but incorporating this driver.

Or...buy the right SFPs in the future...?

Let me know if I can assist in any way.

regards, Ben

On Mon, Feb 18, 2013 at 11:41 AM, Benjeman Meekhof <bmeekhof@umich.edu> wrote: We have run into this problem before and as a result compiled an updated driver for the NIC. Unfortunately I think we only produced a kernel with the updated module and not a module package. We run this kernel on our DYNES FDT machines. I would guess we don't want to run it on the perfsonar boxes since we don't produce the perfsonar distribution and don't want to be maintaining our own flavor of it.

Let me see about producing an ixgbe driver package (RPM) that has the patched driver that you can install on your perfsonar box. We will surely have this problem on other boxes.

-Ben

On Fri, Feb 15, 2013 at 8:03 PM, <siemsen@ucar.edu> wrote: Thanks for the help with the fan issue. I have a new problem. We racked one of the FRGP servers in its home in Denver today (we'll do the other on later). We couldn't get the 10G NIC to work. It got link, but we got this in /var/log/messages:

Feb 15 15:12:40 perfsonar-bw kernel: ixgbe 0000:04:00.0: failed to load because an unsupported SFP+ module type was detected. Feb 15 15:12:40 perfsonar-bw kernel: ixgbe 0000:04:00.0: Reload the driver after installing a supported module.

This is using the Dell SR SFP that you sent us with the machines.

Have you guys seen this? Perhaps we need to recompile a driver or give the driver options or something?

We considered working around this by connecting the server to our Juniper router using a Juniper twinax cable. Unfortunately, the twinax is in Cheyenne, so that option can't be tested until someone in our crew happens to make a trip to Cheyenne. That happens about once a week or so. What do you think of this idea?

-- Pete

arlake228 commented 9 years ago

Comment #1 originally posted by arlake228 on 2013-03-18T16:22:42.000Z:

Assigning to self to see if it builds cleanly. If it doesn't I will likely assign to someone else.

arlake228 commented 9 years ago

Comment #2 originally posted by arlake228 on 2013-04-12T19:31:33.000Z:

<empty>

arlake228 commented 9 years ago

Comment #3 originally posted by arlake228 on 2013-10-18T17:55:28.000Z:

We discussed this and decided that the toolkit should only use default CentOS drivers.

Custom drivers will not be supported.