Ensure we can read out link status from the CTP6 FE at Chamberlin

[ekfriis]

You need to:

1. Compile the softipbus TCP server for Petalinux. See the README here [1] for details.
2. Upload the softipbus-forward server to /tmp on the CTP6 backend. Get the appropriate IP address and ssh password (user is root) from Jes.
3. SSH and run /tmp/soft-ipbus on the BE (leave it running)
4. Compile ctp6_fe_uart_ipbus and upload it (make upload in its directory)
5. Use ctp6commander [2] CLI to check the link status
6. Document these instructions in the README.md

[1] https://svnweb.cern.ch/trac/cactus/browser/trunk/cactuscore/softipbus/README.md [2] https://github.com/uwcms/ctp6commander

[ekfriis]

Jes is working on getting the JTAG XMD going for step 4, the IP address to ssh to BE is ssh root@192.168.1.31

[nwoods]

I must be missing something obvious here -- the first step says to compile with make but... compile what? Where? It looks like the cactuscore-uhal packages are already installed by Yum, but I can't find anything on Ayinger that looks like TCP or SoftIPBus, so I don't even know where to find the makefile.

[nwoods]

Also, how do I get the Xilinx Petalinux SDK? AFAIK it's not installed on Ayinger.

[ekfriis]

You want to run make in the softipbus package, which is on SVN

svn co svn+ssh://svn.cern.ch/reps/cactus/trunk/cactuscore/softipbus$HOME/trigger_code/softipbus

Check out environment.sh in cms-calo-layer1 for setting up the petalinux SDK.

Evan

On Wed, Oct 2, 2013 at 11:30 PM, nwoods notifications@github.com wrote:

Also, how do I get the Xilinx Petalinux SDK? AFAIK it's not installed on Ayinger.

— Reply to this email directly or view it on GitHubhttps://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-25578767 .

[nwoods]

Any time I try to scp onto or off of the CTP6 is fails with the message

sh: scp: Input/output error

I'll bug Jes about it after my afternoon class, but if you have any ideas in the mean time I'd love to hear them.

Nate

[ekfriis]

Can you ssh into it? Are you scp into /tmp?

On Thu, Oct 3, 2013 at 7:30 PM, nwoods notifications@github.com wrote:

Any time I try to scp onto or off of the CTP6 is fails with the message

sh: scp: Input/output error

I'll bug Jes about it after my afternoon class, but if you have any ideas in the mean time I'd love to hear them.

Nate

— Reply to this email directly or view it on GitHubhttps://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-25640844 .

[nwoods]

ssh works fine. I can't scp into /tmp or into ~

[ekfriis]

If you ssh in, can you "touch /tmp/test_creation"

On Thu, Oct 3, 2013 at 7:39 PM, nwoods notifications@github.com wrote:

ssh works fine. I can't scp into /tmp or into ~

— Reply to this email directly or view it on GitHubhttps://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-25641529 .

[nwoods]

Yep, already tried that (it works)... I probably need to ask Jes. On Oct 3, 2013 12:42 PM, "Evan K. Friis" notifications@github.com wrote:

If you ssh in, can you "touch /tmp/test_creation"

On Thu, Oct 3, 2013 at 7:39 PM, nwoods notifications@github.com wrote:

ssh works fine. I can't scp into /tmp or into ~

— Reply to this email directly or view it on GitHub< https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-25641529> .

— Reply to this email directly or view it on GitHubhttps://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-25641848 .

[nwoods]

So fun story: the CTP6 apparently committed circuit board seppuku and corrupted its own root directory. Jes is reflashing Petalinux and with any luck we'll be back in business in a few minutes.

We should probably look into this carefully, though. Jes says that there's almost nothing a person could have done to cause this, so it was probably a board error and we had better make sure it's not going to do that too often.

[nwoods]

I got softipbus and softipbus-forward successfully uploaded onto the CTP6, but neither one will run. I get the following errors (in which I am replacing unreadable characters with 'X'):

~ # /tmp/softipbus
/tmp/softipbus: line 1: syntax error: unexpected word (expecting ")")
~ # /tmp/softipbus-forward 
/tmp/softipbus-forward: line 1: syntax error: unexpected end of file (expecting ")")

Running make with $PETALINUX defined causes the compiler to exit prematurely and tell me that I can only run make unittest (which does not fix the problem), so I tried again without it, uploaded it, and got the same result but with a few extra error messages:

~ # /tmp/softipbus
/tmp/softipbus: line 1: syntax error: unexpected word (expecting ")")
~ # /tmp/softipbus-forward 
/tmp/softipbus-forward: line 1: can't create X
�: Read-only file system
/tmp/softipbus-forward: line 1: XELFXXX

: not found
/tmp/softipbus-forward: line 1: syntax error: unexpected end of file (expecting ")")

[ekfriis]

For it to work on the CTP6, PETALINUX definitely has to be set. The CTP6 has a different architecture than ayinger/your PC so you need to use the special GCC binary from petalinux to "cross-compile" it correctly - otherwise you will expect errors like your second one. The reason for the "error" reported at the end is because since you have cross-compiled to the MB arch you can't run the binaries on your local PC to test them. I've fixed the Makefile so it doesn't cause an error, just an informative message. Run "svn update" in your softipbus directory to get it.

It seems that you are still suffering from the same problem (wrong target architecture [1]), even with $PETALINUX set. I think this is because @jtikalsky has changed the PETALINUX distribution here [2] to the ZYNQ chip. Jes, can you have a look? The $CC that I pick up from include user-commons.mk has changed from "microblazeel-xilinx-linux-gnu-gcc" to "arm-xilinx-linux-gnueabi-gcc".

[1] http://gaoithe.livejournal.com/33519.html [2] /afs/hep.wisc.edu/home/uwhepfpga/petalinux-v12.12-final-full/software/petalinux-dist/.config

[nwoods]

OK, I'm working on UCT energy sum rates and efficiencies in the hope of having them for the JETMET talk on Tuesday. @jtikalsky, could you let me know when you figure this out? Thanks!

[jtikalsky]

@nwoods, @ekfriis has the right idea, I'm betting. You cannot use this petalinux installation for your work at this time, as it is being used for a build for the ZC702 board. You will need to set up a separate environment to do your builds in, or risk this problem recurring frequently even if I do switch modes back for you right now.

@ekfriis, I believe you did something like this at CERN for yourself when you were developing softipbus initially. Can you give a list of steps you had to do to get that working? I know what's required for a full ground-up build, but a simplified procedure to get to the point of cross-compilation may be more useful here.

@nwoods, for PetaLinux licensing, if you are on the Chamberlin network you can set the environment variable

XILINXD_LICENSE_FILE='2100@dawn.physics.wisc.edu'

which will take care of everything.

[jtikalsky]

So testing Zen mode is interesting

test

testing oranges and pies.
self post.

[ekfriis]

Hi @jtikalsky,

I found an old makefile where I manually pick out the GCC binary from the petalinux folder:

https://github.com/uwcms/soft-ipbus/blob/68ddc9bd2b89c181c460b61df39902bf3b2e7335/Makefile.petalinux#L14

Hopefully @nwoods will find it useful. In any case, can we just have a separate PETALINUX directory with the CTP6? I think this would be easiest.

[nwoods]

I got ipbus and ipbus-forward running on the CTP6 (I think -- they both just sit silently until killed), and I was able, after some effort, to get ctp6_fe_uart_ipbus compiled. At the end of the compilation, it gives an error, saying that the CTP cable is not connected:

echo "connect mb mdm; dow ctp6_fe_uart_ipbus.elf;" | xmd
Xilinx Microprocessor Debugger (XMD) Engine
Xilinx EDK 14.4 Build EDK_P.49d
Copyright (c) 1995-2012 Xilinx, Inc.  All rights reserved.

XMD% 
XMD% ERROR: Failed to Open JTAG Cable
    Cable target is not connected to the host

XMD% 

Since I can ssh into the CTP6, this seems unlikely. Any idea what this could be?

[ekfriis]

Are you on Ayinger? The SSH connection goes over the ethernet netwrok, and is different than the JTAG - the JTAG is a USB dongle thing which plugs directly into the board and lets you do fancy HW stuff.

Look and see if a red JTAG programmer is plugged into Ayinger and into the CTP6. If not ask Jes/Tom to show you how to connect. Sometime the driver gets roached/wedged as Tom/Jes would say, supposedly

sudo chmod a+rw /dev/windrvr6

might fix it. If not you need to ask Jes, she knows how to do it. We don't understand why it stops working sometimes.

On Wed, Oct 9, 2013 at 12:25 AM, nwoods notifications@github.com wrote:

I got ipbus and ipbus-forward running on the CTP6 (I think -- they both just sit silently until killed), and I was able, after some effort, to get ctp6_fe_uart_ipbus compiled. At the end of the compilation, it gives an error, saying that the CTP cable is not connected:

echo "connect mb mdm; dow ctp6_fe_uart_ipbus.elf;" | xmd Xilinx Microprocessor Debugger (XMD) Engine Xilinx EDK 14.4 Build EDK_P.49d Copyright (c) 1995-2012 Xilinx, Inc. All rights reserved.

XMD% XMD% ERROR: Failed to Open JTAG Cable Cable target is not connected to the host

XMD%

Since I can ssh into the CTP6, this seems unlikely. Any idea what this could be?

— Reply to this email directly or view it on GitHubhttps://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-25933051 .

[nwoods]

I'm about 80% sure that I was just able to compile ctp6_fe_uart_ipbus and upload it. The 20% uncertainty comes from the fact that I actually built and uploaded something called payload.elf, but as far as I could tell from the makefile, this is the correct thing.

I can't finish putting all the pieces together now, though, because the card's TCP connection seems to be down. I told @jtikalsky and she said she'd look at it when she got a chance. Jes, can you email me or comment here when you know what's up? Thanks!

[ekfriis]

Yes, take back your 20% - before each binary (.elf) had a different name, I made them all build to payload.elf now to make transporting xmd commands a bit easier.

On Wed, Oct 9, 2013 at 6:41 PM, nwoods notifications@github.com wrote:

I'm about 80% sure that I was just able to compile ctp6_fe_uart_ipbus and upload it. The 20% uncertainty comes from the fact that I actually built and uploaded something called payload.elf, but as far as I could tell from the makefile, this is the correct thing.

I can't finish putting all the pieces together now, though, because the card's TCP connection seems to be down. I told @jtikalskyhttps://github.com/jtikalskyand she said she'd look at it when she got a chance. Jes, can you email me or comment here when you know what's up? Thanks!

— Reply to this email directly or view it on GitHubhttps://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-25986923 .

[nwoods]

Progress: ctp6_fe_uart_ipbus/payload.elf will make upload without complaining, and both softipbus and softipbus-forward will run on the CTP6 without giving error messages.

I'm close to get ctp6commander working. I can't source the custom version of Python because Ayinger can't see cvmfs. Jes tells me that we might be able to get cvmfs on Ayinger but if there's another way to get it "without serious trouble", that would be preferable. @ekfriis, do you know if that's possible?

On Wed, Oct 9, 2013 at 11:56 AM, Evan K. Friis notifications@github.comwrote:

Yes, take back your 20% - before each binary (.elf) had a different name, I made them all build to payload.elf now to make transporting xmd commands a bit easier.

On Wed, Oct 9, 2013 at 6:41 PM, nwoods notifications@github.com wrote:

I'm about 80% sure that I was just able to compile ctp6_fe_uart_ipbus and upload it. The 20% uncertainty comes from the fact that I actually built and uploaded something called payload.elf, but as far as I could tell from the makefile, this is the correct thing.

I can't finish putting all the pieces together now, though, because the card's TCP connection seems to be down. I told @jtikalsky< https://github.com/jtikalsky>and she said she'd look at it when she got a chance. Jes, can you email me or comment here when you know what's up? Thanks!

— Reply to this email directly or view it on GitHub< https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-25986923> .

— Reply to this email directly or view it on GitHubhttps://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-25988292 .

[jtikalsky]

What is special about this version?

On 10/10/2013 04:47 PM, nwoods wrote:

Progress: ctp6_fe_uart_ipbus/payload.elf will make upload without complaining, and both softipbus and softipbus-forward will run on the CTP6 without giving error messages.

I'm close to get ctp6commander working. I can't source the custom version of Python because Ayinger can't see cvmfs. Jes tells me that we might be able to get cvmfs on Ayinger but if there's another way to get it "without serious trouble", that would be preferable. @ekfriis, do you know if that's possible?

On Wed, Oct 9, 2013 at 11:56 AM, Evan K. Friis notifications@github.comwrote:

Yes, take back your 20% - before each binary (.elf) had a different name, I made them all build to payload.elf now to make transporting xmd commands a bit easier.

On Wed, Oct 9, 2013 at 6:41 PM, nwoods notifications@github.com wrote:

I'm about 80% sure that I was just able to compile ctp6_fe_uart_ipbus and upload it. The 20% uncertainty comes from the fact that I actually built and uploaded something called payload.elf, but as far as I could tell from the makefile, this is the correct thing.

I can't finish putting all the pieces together now, though, because the card's TCP connection seems to be down. I told @jtikalsky< https://github.com/jtikalsky>and she said she'd look at it when she got a chance. Jes, can you email me or comment here when you know what's up? Thanks!

— Reply to this email directly or view it on GitHub< https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-25986923> .

— Reply to this email directly or view it on GitHubhttps://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-25988292

.

— Reply to this email directly or view it on GitHub https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26094656.

[nwoods]

I have no idea, but it seems that it's the only one that will run https://github.com/uwcms/ctp6commander correctly. (See the readme at that link).

On Thu, Oct 10, 2013 at 4:48 PM, jtikalsky notifications@github.com wrote:

What is special about this version?

On 10/10/2013 04:47 PM, nwoods wrote:

Progress: ctp6_fe_uart_ipbus/payload.elf will make upload without complaining, and both softipbus and softipbus-forward will run on the CTP6 without giving error messages.

I'm close to get ctp6commander working. I can't source the custom version of Python because Ayinger can't see cvmfs. Jes tells me that we might be able to get cvmfs on Ayinger but if there's another way to get it "without serious trouble", that would be preferable. @ekfriis, do you know if that's possible?

On Wed, Oct 9, 2013 at 11:56 AM, Evan K. Friis notifications@github.comwrote:

Yes, take back your 20% - before each binary (.elf) had a different name, I made them all build to payload.elf now to make transporting xmd commands a bit easier.

On Wed, Oct 9, 2013 at 6:41 PM, nwoods notifications@github.com wrote:

I'm about 80% sure that I was just able to compile ctp6_fe_uart_ipbus and upload it. The 20% uncertainty comes from the fact that I actually built and uploaded something called payload.elf, but as far as I could tell from the makefile, this is the correct thing.

I can't finish putting all the pieces together now, though, because the card's TCP connection seems to be down. I told @jtikalsky< https://github.com/jtikalsky>and she said she'd look at it when she got a chance. Jes, can you email me or comment here when you know what's up? Thanks!

— Reply to this email directly or view it on GitHub<

https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-25986923>

.

— Reply to this email directly or view it on GitHub< https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-25988292>

.

— Reply to this email directly or view it on GitHub <https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26094656 .

— Reply to this email directly or view it on GitHubhttps://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26094744 .

[ekfriis]

The default version of python is 2.4, which was not compatible with the dependencies, which is why I used the CVMFS version.

However, there is a 2.6 version installed on ayinger.

I just pushed ac8ffe559698ba5960f93b165bd0d06911ac3b26, which make setup_venv.sh work out of the box on ayinger (and have removed the obsolete CVMFS instruction from the README).

BTW Nate, the code in ctp6commander is a not-completely-tested refactoring of the original script here [1] used for the integration test. It should work, but if you have troubles please try the softipbus/scripts version as well.

[1] https://svnweb.cern.ch/trac/cactus/browser/trunk/cactuscore/softipbus/scripts/ctp6

On Thu, Oct 10, 2013 at 11:51 PM, nwoods notifications@github.com wrote:

I have no idea, but it seems that it's the only one that will run https://github.com/uwcms/ctp6commander correctly. (See the readme at that link).

On Thu, Oct 10, 2013 at 4:48 PM, jtikalsky notifications@github.com wrote:

What is special about this version?

On 10/10/2013 04:47 PM, nwoods wrote:

Progress: ctp6_fe_uart_ipbus/payload.elf will make upload without complaining, and both softipbus and softipbus-forward will run on the CTP6 without giving error messages.

I'm close to get ctp6commander working. I can't source the custom version of Python because Ayinger can't see cvmfs. Jes tells me that we might be able to get cvmfs on Ayinger but if there's another way to get it "without serious trouble", that would be preferable. @ekfriis, do you know if that's possible?

On Wed, Oct 9, 2013 at 11:56 AM, Evan K. Friis notifications@github.comwrote:

Yes, take back your 20% - before each binary (.elf) had a different name, I made them all build to payload.elf now to make transporting xmd commands a bit easier.

On Wed, Oct 9, 2013 at 6:41 PM, nwoods notifications@github.com wrote:

I'm about 80% sure that I was just able to compile ctp6_fe_uart_ipbus and upload it. The 20% uncertainty comes from the fact that I actually built and uploaded something called payload.elf, but as far as I could tell from the makefile, this is the correct thing.

I can't finish putting all the pieces together now, though, because the card's TCP connection seems to be down. I told @jtikalsky< https://github.com/jtikalsky>and she said she'd look at it when she got a chance. Jes, can you email me or comment here when you know what's up? Thanks!

— Reply to this email directly or view it on GitHub<

https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-25986923>

.

— Reply to this email directly or view it on GitHub< https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-25988292>

.

— Reply to this email directly or view it on GitHub < https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26094656 .

— Reply to this email directly or view it on GitHub< https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26094744> .

— Reply to this email directly or view it on GitHubhttps://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26094887 .

[nwoods]

Since Evan told me how to make xmd talk to the correct FPGA it's not crashing any more, but nothing I try to do with xmd is working, saying that the processor is stalled. Here's the command and error message:

XMD% connect mb mdm -debugdevice deviceNr 2

JTAG chain configuration
--------------------------------------------------
Device   ID Code        IR Length    Part Name
 1       042a2093          10        XC6VHX250T
 2       442a8093          10        XC6VHX380T

MicroBlaze Processor Configuration :
-------------------------------------
Version............................8.40.b
Optimization.......................Performance
Interconnect.......................AXI-LE
MMU Type...........................No_MMU
No of PC Breakpoints...............2
No of Read Addr/Data Watchpoints...0
No of Write Addr/Data Watchpoints..0
Instruction Cache Support..........on
Instruction Cache Base Address.....0xf0000000
Instruction Cache High Address.....0xffffffff
Data Cache Support.................on
Data Cache Base Address............0xf0000000
Data Cache High Address............0xffffffff
Exceptions  Support................on
FPU  Support.......................off
Hard Divider Support...............off
Hard Multiplier Support............on - (Mul32)
Barrel Shifter Support.............off
MSR clr/set Instruction Support....on
Compare Instruction Support........on
Data Cache Write-back Support......off
Fault Tolerance Support............off
Stack Protection Support...........off
Processor Could not be STOPPED - MicroBlaze Pipeline Stalled on a Blocking
Instruction or Invalid Bus Access
Stalled PC: 0x3ffffffc
Try Resetting the Processor to Continue..Processor is stalled at address
0x2d2d0a3e. UNABLE to STOP MicroBlaze
 Debug Operation requires Processor in STOP State.
1. Try to reset the Processor and check if the Processor is Stopped
2. Check your System Design for Correctness

Connected to "mb" target. id = 0
Starting GDB server for "mb" target (id = 0) at TCP port no 1234

This may be something obvious that I don't know because I'm still new to this. Jes, will you be around after lunch to take a look?

On Fri, Oct 11, 2013 at 2:52 AM, Evan K. Friis notifications@github.comwrote:

The default version of python is 2.4, which was not compatible with the dependencies, which is why I used the CVMFS version.

However, there is a 2.6 version installed on ayinger.

I just pushed ac8ffe559698ba5960f93b165bd0d06911ac3b26, which make setup_venv.sh work out of the box on ayinger (and have removed the obsolete CVMFS instruction from the README).

BTW Nate, the code in ctp6commander is a not-completely-tested refactoring of the original script here [1] used for the integration test. It should work, but if you have troubles please try the softipbus/scripts version as well.

[1]

https://svnweb.cern.ch/trac/cactus/browser/trunk/cactuscore/softipbus/scripts/ctp6

On Thu, Oct 10, 2013 at 11:51 PM, nwoods notifications@github.com wrote:

I have no idea, but it seems that it's the only one that will run https://github.com/uwcms/ctp6commander correctly. (See the readme at that link).

On Thu, Oct 10, 2013 at 4:48 PM, jtikalsky notifications@github.com wrote:

What is special about this version?

On 10/10/2013 04:47 PM, nwoods wrote:

Progress: ctp6_fe_uart_ipbus/payload.elf will make upload without complaining, and both softipbus and softipbus-forward will run on the CTP6 without giving error messages.

I'm close to get ctp6commander working. I can't source the custom version of Python because Ayinger can't see cvmfs. Jes tells me that we might be able to get cvmfs on Ayinger but if there's another way to get it "without serious trouble", that would be preferable. @ekfriis, do you know if that's possible?

On Wed, Oct 9, 2013 at 11:56 AM, Evan K. Friis notifications@github.comwrote:

Yes, take back your 20% - before each binary (.elf) had a different name, I made them all build to payload.elf now to make transporting xmd commands a bit easier.

On Wed, Oct 9, 2013 at 6:41 PM, nwoods notifications@github.com wrote:

I'm about 80% sure that I was just able to compile ctp6_fe_uart_ipbus and upload it. The 20% uncertainty comes from the fact that I actually built and uploaded something called payload.elf, but as far as I could tell from the makefile, this is the correct thing.

I can't finish putting all the pieces together now, though, because the card's TCP connection seems to be down. I told @jtikalsky< https://github.com/jtikalsky>and she said she'd look at it when she got a chance. Jes, can you email me or comment here when you know what's up? Thanks!

— Reply to this email directly or view it on GitHub<

https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-25986923>

.

— Reply to this email directly or view it on GitHub<

https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-25988292>

.

— Reply to this email directly or view it on GitHub < https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26094656 .

— Reply to this email directly or view it on GitHub< https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26094744>

.

— Reply to this email directly or view it on GitHub< https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26094887> .

— Reply to this email directly or view it on GitHubhttps://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26120167 .

[ekfriis]

Try typing "rst" to reset it. Then "dow" the program again, then "run" it

On Fri, Oct 11, 2013 at 7:02 PM, nwoods notifications@github.com wrote:

Since Evan told me how to make xmd talk to the correct FPGA it's not crashing any more, but nothing I try to do with xmd is working, saying that the processor is stalled. Here's the command and error message:

XMD% connect mb mdm -debugdevice deviceNr 2

JTAG chain configuration
--------------------------------------------------
Device ID Code IR Length Part Name
1 042a2093 10 XC6VHX250T
2 442a8093 10 XC6VHX380T

MicroBlaze Processor Configuration :
-------------------------------------
Version............................8.40.b
Optimization.......................Performance
Interconnect.......................AXI-LE
MMU Type...........................No_MMU
No of PC Breakpoints...............2
No of Read Addr/Data Watchpoints...0
No of Write Addr/Data Watchpoints..0
Instruction Cache Support..........on
Instruction Cache Base Address.....0xf0000000
Instruction Cache High Address.....0xffffffff
Data Cache Support.................on
Data Cache Base Address............0xf0000000
Data Cache High Address............0xffffffff
Exceptions Support................on
FPU Support.......................off
Hard Divider Support...............off
Hard Multiplier Support............on - (Mul32)
Barrel Shifter Support.............off
MSR clr/set Instruction Support....on
Compare Instruction Support........on
Data Cache Write-back Support......off
Fault Tolerance Support............off
Stack Protection Support...........off
Processor Could not be STOPPED - MicroBlaze Pipeline Stalled on a Blocking
Instruction or Invalid Bus Access
Stalled PC: 0x3ffffffc
Try Resetting the Processor to Continue..Processor is stalled at address
0x2d2d0a3e. UNABLE to STOP MicroBlaze
Debug Operation requires Processor in STOP State.
1. Try to reset the Processor and check if the Processor is Stopped
2. Check your System Design for Correctness

Connected to "mb" target. id = 0
Starting GDB server for "mb" target (id = 0) at TCP port no 1234

This may be something obvious that I don't know because I'm still new to this. Jes, will you be around after lunch to take a look?

On Fri, Oct 11, 2013 at 2:52 AM, Evan K. Friis notifications@github.comwrote:

The default version of python is 2.4, which was not compatible with the dependencies, which is why I used the CVMFS version.

However, there is a 2.6 version installed on ayinger.

I just pushed ac8ffe559698ba5960f93b165bd0d06911ac3b26, which make setup_venv.sh work out of the box on ayinger (and have removed the obsolete CVMFS instruction from the README).

BTW Nate, the code in ctp6commander is a not-completely-tested refactoring of the original script here [1] used for the integration test. It should work, but if you have troubles please try the softipbus/scripts version as well.

[1]

https://svnweb.cern.ch/trac/cactus/browser/trunk/cactuscore/softipbus/scripts/ctp6

On Thu, Oct 10, 2013 at 11:51 PM, nwoods notifications@github.com wrote:

I have no idea, but it seems that it's the only one that will run https://github.com/uwcms/ctp6commander correctly. (See the readme at that link).

On Thu, Oct 10, 2013 at 4:48 PM, jtikalsky notifications@github.com wrote:

What is special about this version?

On 10/10/2013 04:47 PM, nwoods wrote:

Progress: ctp6_fe_uart_ipbus/payload.elf will make upload without complaining, and both softipbus and softipbus-forward will run on the CTP6 without giving error messages.

I'm close to get ctp6commander working. I can't source the custom version of Python because Ayinger can't see cvmfs. Jes tells me that we might be able to get cvmfs on Ayinger but if there's another way to get it "without serious trouble", that would be preferable. @ekfriis, do you know if that's possible?

On Wed, Oct 9, 2013 at 11:56 AM, Evan K. Friis notifications@github.comwrote:

Yes, take back your 20% - before each binary (.elf) had a different name, I made them all build to payload.elf now to make transporting xmd commands a bit easier.

On Wed, Oct 9, 2013 at 6:41 PM, nwoods notifications@github.com

wrote:

I'm about 80% sure that I was just able to compile ctp6_fe_uart_ipbus and upload it. The 20% uncertainty comes from the fact that I actually built and uploaded something called payload.elf, but as far as I could tell from the makefile, this is the correct thing.

I can't finish putting all the pieces together now, though, because the card's TCP connection seems to be down. I told @jtikalsky< https://github.com/jtikalsky>and she said she'd look at it when she got a chance. Jes, can you email me or comment here when you know what's up? Thanks!

— Reply to this email directly or view it on GitHub<

https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-25986923>

.

— Reply to this email directly or view it on GitHub<

https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-25988292>

.

— Reply to this email directly or view it on GitHub <

https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26094656

.

— Reply to this email directly or view it on GitHub<

https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26094744>

.

— Reply to this email directly or view it on GitHub< https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26094887>

.

— Reply to this email directly or view it on GitHub< https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26120167> .

— Reply to this email directly or view it on GitHubhttps://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26153511 .

[nwoods]

rst worked, and it looks like it's downloading and running successfully. I upgraded ctp6commander but now when I ./setup_venv.sh I get the following (presumably python version-related?) error:

nwoods@ayinger /afs/hep.wisc.edu/cms/nwoods/ctp6commander  master$ ./setup_venv.sh --upgrade
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1294k  100 1294k    0     0  1984k      0 --:--:-- --:--:-- --:--:-- 3052k
Using real prefix '/usr'
New python executable in env/bin/python2.6
Traceback (most recent call last):
  File "virtualenv-1.10.1/virtualenv.py", line 2308, in <module>
    main()
  File "virtualenv-1.10.1/virtualenv.py", line 821, in main
    symlink=options.symlink)
  File "virtualenv-1.10.1/virtualenv.py", line 956, in create_environment
    site_packages=site_packages, clear=clear, symlink=symlink))
  File "virtualenv-1.10.1/virtualenv.py", line 1247, in install_python
    shutil.copyfile(executable, py_executable)
  File "/usr/lib64/python2.6/shutil.py", line 48, in copyfile
    raise Error("`%s` and `%s` are the same file" % (src, dst))
shutil.Error: `/afs/hep.wisc.edu/cms/nwoods/ctp6commander/env/bin/python2.6` and `env/bin/python2.6` are the same file
Requirement already satisfied (use --upgrade to upgrade): argparse in ./env/lib/python2.6/site-packages
Cleaning up...
Requirement already satisfied (use --upgrade to upgrade): flask in ./env/lib/python2.6/site-packages
Requirement already satisfied (use --upgrade to upgrade): Werkzeug>=0.7 in ./env/lib/python2.6/site-packages (from flask)
Requirement already satisfied (use --upgrade to upgrade): Jinja2>=2.4 in ./env/lib/python2.6/site-packages (from flask)
Requirement already satisfied (use --upgrade to upgrade): itsdangerous>=0.21 in ./env/lib/python2.6/site-packages (from flask)
Requirement already satisfied (use --upgrade to upgrade): markupsafe in ./env/lib/python2.6/site-packages (from Jinja2>=2.4->flask)
Cleaning up...
Requirement already satisfied (use --upgrade to upgrade): termcolor in ./env/lib/python2.6/site-packages
Cleaning up...

[nwoods]

I was able to fix the python issue by editing virtualenv.py to ignore the stupid exception it was throwing (it even has a FIXME comment in it about this...). Now when I try to run cli.py, it fails with the error

nwoods@ayinger /afs/hep.wisc.edu/cms/nwoods/ctp6commander  master$ python cli.py 
Traceback (most recent call last):
  File "cli.py", line 12, in <module>
    import uhal
  File "/afs/hep.wisc.edu/cms/nwoods/cactus/trunk/cactuscore/uhal/pycohal/pkg/uhal/__init__.py", line 1, in <module>
    from _core import *
ImportError: ../cactus/trunk/cactuscore/extern/boost/RPMBUILD/SOURCES/lib/libboost_python.so.1.48.0: undefined symbol: Py_InitModule4

[ekfriis]

Have you executed the "source environment.sh" from cms-calo-layer1?

On Fri, Oct 11, 2013 at 8:54 PM, nwoods notifications@github.com wrote:

I was able to fix the python issue by editing virtualenv.py to ignore the stupid exception it was throwing (it even has a FIXME comment in it about this...). Now when I try to run cli.py, it fails with the error

nwoods@ayinger /afs/hep.wisc.edu/cms/nwoods/ctp6commander master$ python cli.py Traceback (most recent call last): File "cli.py", line 12, in import uhal File "/afs/hep.wisc.edu/cms/nwoods/cactus/trunk/cactuscore/uhal/pycohal/pkg/uhal/init.py", line 1, in from _core import * ImportError: ../cactus/trunk/cactuscore/extern/boost/RPMBUILD/SOURCES/lib/libboost_python.so.1.48.0: undefined symbol: Py_InitModule4

— Reply to this email directly or view it on GitHubhttps://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26162311 .

[nwoods]

Yes.

On Fri, Oct 11, 2013 at 2:06 PM, Evan K. Friis notifications@github.comwrote:

Have you executed the "source environment.sh" from cms-calo-layer1?

On Fri, Oct 11, 2013 at 8:54 PM, nwoods notifications@github.com wrote:

I was able to fix the python issue by editing virtualenv.py to ignore the stupid exception it was throwing (it even has a FIXME comment in it about this...). Now when I try to run cli.py, it fails with the error

nwoods@ayinger /afs/hep.wisc.edu/cms/nwoods/ctp6commander master$ python cli.py Traceback (most recent call last): File "cli.py", line 12, in import uhal File "/afs/ hep.wisc.edu/cms/nwoods/cactus/trunk/cactuscore/uhal/pycohal/pkg/uhal/init.py", line 1, in from _core import * ImportError: ../cactus/trunk/cactuscore/extern/boost/RPMBUILD/SOURCES/lib/libboost_python.so.1.48.0: undefined symbol: Py_InitModule4

— Reply to this email directly or view it on GitHub< https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26162311> .

— Reply to this email directly or view it on GitHubhttps://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26163288 .

[ekfriis]

Crap, it seems the cactus repo is compiled against the python 2.4 version that is default on the system. Ask Jes if she knows a way to make the default python version the 2.6 on the machine (or for you) then recompile the stuff in ../cactus/trunk

On Fri, Oct 11, 2013 at 9:11 PM, nwoods notifications@github.com wrote:

Yes.

On Fri, Oct 11, 2013 at 2:06 PM, Evan K. Friis notifications@github.comwrote:

Have you executed the "source environment.sh" from cms-calo-layer1?

On Fri, Oct 11, 2013 at 8:54 PM, nwoods notifications@github.com wrote:

I was able to fix the python issue by editing virtualenv.py to ignore the stupid exception it was throwing (it even has a FIXME comment in it about this...). Now when I try to run cli.py, it fails with the error

nwoods@ayinger /afs/hep.wisc.edu/cms/nwoods/ctp6commander master$ python cli.py Traceback (most recent call last): File "cli.py", line 12, in import uhal File "/afs/

hep.wisc.edu/cms/nwoods/cactus/trunk/cactuscore/uhal/pycohal/pkg/uhal/init.py",

line 1, in

from _core import * ImportError:

../cactus/trunk/cactuscore/extern/boost/RPMBUILD/SOURCES/lib/libboost_python.so.1.48.0:

undefined symbol: Py_InitModule4

— Reply to this email directly or view it on GitHub< https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26162311>

.

— Reply to this email directly or view it on GitHub< https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26163288> .

— Reply to this email directly or view it on GitHubhttps://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26163675 .

[jtikalsky]

In case of confusion, I believe @nwoods and I fixed this earlier. It was a case of the env/ directory in the ctp6commander needing to be cleaned up, regenerated with your new code, and then the other stuff recompiled, @ekfriis

On 10/11/2013 02:18 PM, Evan K. Friis wrote:

Crap, it seems the cactus repo is compiled against the python 2.4 version that is default on the system. Ask Jes if she knows a way to make the default python version the 2.6 on the machine (or for you) then recompile the stuff in ../cactus/trunk

On Fri, Oct 11, 2013 at 9:11 PM, nwoods notifications@github.com wrote:

Yes.

On Fri, Oct 11, 2013 at 2:06 PM, Evan K. Friis notifications@github.comwrote:

Have you executed the "source environment.sh" from cms-calo-layer1?

On Fri, Oct 11, 2013 at 8:54 PM, nwoods notifications@github.com wrote:

I was able to fix the python issue by editing virtualenv.py to ignore the stupid exception it was throwing (it even has a FIXME comment in it about this...). Now when I try to run cli.py, it fails with the error

nwoods@ayinger /afs/hep.wisc.edu/cms/nwoods/ctp6commander master$ python cli.py Traceback (most recent call last): File "cli.py", line 12, in import uhal File "/afs/

hep.wisc.edu/cms/nwoods/cactus/trunk/cactuscore/uhal/pycohal/pkg/uhal/init.py",

line 1, in

from _core import * ImportError:

../cactus/trunk/cactuscore/extern/boost/RPMBUILD/SOURCES/lib/libboost_python.so.1.48.0:

undefined symbol: Py_InitModule4

— Reply to this email directly or view it on GitHub<

https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26162311>

.

— Reply to this email directly or view it on GitHub< https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26163288> .

— Reply to this email directly or view it on GitHubhttps://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26163675

.

— Reply to this email directly or view it on GitHub https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26164238.

[nwoods]

@ekfriis, I think at this point the only thing left is to run cli.py. It's not crashing because of Python any more, but it wants inputs that I don't know how to give it. Specifically, links and connections.

Do I understand right that the connections are in the form of an xml file? Does one exist, or do I need to make one based on the examples in the uhal code? If a connection is not specified, cli.py expects an os variable $CTP6_CONNECTION, but none of the configuration files set one. Once we have that, we'll need another xml file with the addresses; do we have one of those? Or does Matthias?

Once I've got that, I'm still not sure how to specify a link. Is that a number like the 60001 and 60002s I've seen a few places?

[ekfriis]

The port is defined int he xml connect. See softipbus/etc/ctp6_fe.xml for on which should work for the CTP6. (This xml should be moved to ctp6commander/)

Evan

On Sat, Oct 12, 2013 at 12:42 AM, nwoods notifications@github.com wrote:

@ekfriis https://github.com/ekfriis, I think at this point the only thing left is to run cli.py. It's not crashing because of Python any more, but it wants inputs that I don't know how to give it. Specifically, links and connections.

Do I understand right that the connections are in the form of an xml file? Does one exist, or do I need to make one based on the examples in the uhal code? If a connection is not specified, cli.py expects an os variable $CTP6_CONNECTION, but none of the configuration files set one. Once we have that, we'll need another xml file with the addresses; do we have one of those? Or does Matthias?

Once I've got that, I'm still not sure how to specify a link. Is that a number like the 60001 and 60002s I've seen a few places?

— Reply to this email directly or view it on GitHubhttps://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26180145 .

[nwoods]

OK, I've got cli.py running after fixing a few bugs in it and api.py, but it's definitely not doing what I want it to.

First, I haven't figured out how to correctly specify the specific links I want to test. Right now I'm running over all the links with the command

python cli.py --connection=ctp6_connections.xml status

and it runs, but if I try to specify the links it gives me the error message

ValueError: invalid literal for int() with base 10: <whatever I typed in>

I think I just don't know the correct format, but I've tried enough possibilities that it might just be broken in some way I couldn't figure out.

More importantly, it does not give the link status: it just prints a few horizontal lines and the link numbers and nothing else. I think the issue is in api.py in the following code block

def status(hw, links):
    """ Query the status of the given links.

    Returns a dictionary mapping each link to a STATUS_FLAG state dictionary.

    """
    output = collections.defaultdict(list)
    for flag, flag_cfg in output.iteritems():
        flag_cfg['banks'] = [flag_cfg['prefix'] + bank for bank in BANKS]
        flag_cfg['nodes'] = [hw.getNode(bank) for bank in flag_cfg['banks']]
        flag_cfg['values'] = [node.read() for node in flag_cfg['nodes']]
        # eventually do the whole dispatch all at once
        hw.dispatch()
        for link in links:
            value = bool(flag_cfg['values'][link / 12].value()
                         & (1 << (link % 12)))
            output[link].append((flag, (value != flag_cfg['bad'])))
    return output

I changed the first line of actual code from output = collections.defaultdict(list) because that was throwing an error, but I think this is correct. Unfortunately, I'm not sure exactly what this code is doing, so I don't know how to fix it. It does seem to iterate over an empty dictionary, though, which is confusing to me... When I print the output it gives, it tells me it's returning defaultdict(<type 'list'>, {}).

I'll keep fighting with it tomorrow or on Monday, but right now it's almost 3 a.m. on a Sunday...

[ekfriis]

I think you want ctp6_fe.xml, not connections.xml (feel free to delete the old one).

That loop is defintely wrong, it is always iterating over an empty dictionary.

What error does it throw? Can you paste the whole traceback?

Maybe you should give it a shot w/ the original program in softipbus/scripts and see if you can make that work - then figure out how to fix this one by looking at the differences.

On Sun, Oct 13, 2013 at 9:40 AM, nwoods notifications@github.com wrote:

OK, I've got cli.py running after fixing a few bugs in it and api.py, but it's definitely not doing what I want it to.

First, I haven't figured out how to correctly specify the specific links I want to test. Right now I'm running over all the links with the command

python cli.py --connection=ctp6_connections.xml status

and it runs, but if I try to specify the links it gives me the error message

ValueError: invalid literal for int() with base 10:

I think I just don't know the correct format, but I've tried enough possibilities that it might just be broken in some way I couldn't figure out.

More importantly, it does not give the link status: it just prints a few horizontal lines and the link numbers and nothing else. I think the issue is in api.py in the following code block

def status(hw, links): """ Query the status of the given links.

Returns a dictionary mapping each link to a STATUS_FLAG state dictionary.

"""
output = collections.defaultdict(list)
for flag, flag_cfg in output.iteritems():
    flag_cfg['banks'] = [flag_cfg['prefix'] + bank for bank in BANKS]
    flag_cfg['nodes'] = [hw.getNode(bank) for bank in flag_cfg['banks']]
    flag_cfg['values'] = [node.read() for node in flag_cfg['nodes']]
    # eventually do the whole dispatch all at once
    hw.dispatch()
    for link in links:
        value = bool(flag_cfg['values'][link / 12].value()
                     & (1 << (link % 12)))
        output[link].append((flag, (value != flag_cfg['bad'])))
return output

I changed the first line of actual code from output = collections.defaultdict(list) because that was throwing an error, but I think this is correct. Unfortunately, I'm not sure exactly what this code is doing, so I don't know how to fix it. It does seem to iterate over an empty dictionary, though, which is confusing to me... When I print the output it gives, it tells me it's returning defaultdict(<type 'list'>, {}) .

— Reply to this email directly or view it on GitHubhttps://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26213038 .

[ekfriis]

@nwoods

cc @dabelknap @jtikalsky

Hi, we are now working on this same task at 904. @nwoods, can you send us the modifications to the makefile to make it work? @jtikalsky is there a simple way to build the petalinux config for microblaze so the "correct" way of building works as well?

[ekfriis]

Also, what do I need for a liscense ?

[cc @dabelknap]

[ekfriis]

@nwoods

Sorry, I steered you wrong when I said you did not need ctp6_connections.xml. That is the correct file to use.

Evan

[ekfriis]

@nwoods you can disregard my request for your modifications, I figured it out. I've put the Makefile workarounds into the SVN version of softipbus, see this revision [1]. It should work out of the box at 904 and Chamberlin now.

[1] https://svnweb.cern.ch/trac/cactus/changeset?old=23716%40trunk%2Fcactuscore%2Fsoftipbus%2FMakefile&new=23716%40trunk%2Fcactuscore%2Fsoftipbus%2FMakefile

[ekfriis]

PS @nwoods, I found this twiki that I wrote and forgot about:

https://twiki.cern.ch/twiki/bin/view/CMS/ORSCCTP6Integration#Running_the_monitor

It might be helpful, @dabelknap and I are updating it to fix things that have changed as we go.

[nwoods]

I fixed the biggest few bugs in ctp6commander cli.py and api.py, and it gets a lot further now than before. Now it fails at the line hw.dispatch() (where hw is ctp6.frontend) with the message

nwoods@ayinger /afs/hep.wisc.edu/cms/nwoods/ctp6commander  master$ python
cli.py --connection ctp6_connections.xml status
14-10-13 13:45:31.944377 [0x1f6898f0] WARNING - Closing Socket since
exception detected.
Traceback (most recent call last):
  File "cli.py", line 167, in <module>
    commands[args.command](hw, args)
  File "cli.py", line 48, in do_status
    status_flags = api.status(hw, args.links)
  File "/afs/hep.wisc.edu/cms/nwoods/ctp6commander/api.py", line 156, in
status
    hw.dispatch()
uhal._core.exception:
I'm terribly sorry to have to tell you this, but it appears that there was
an exception:
 * Exception type: uhal::exception::IPbus2PacketHeaderMismatch
 * Description: Exception class to handle the case where the IPbus 2.0
packet header does not match that sent.
 * Additional Information:
   > Returned Packet Header 0x00000000 does not match that sent 0x200000F0
 * Exception occured in thread with ID: 0x1f6a8bf0
 * Current thread has ID: 0x1f6898f0
 * Exception constructed at time:              2013-10-14 13:45:31.944187
 * Exception's what() function called at time: 2013-10-14 13:45:31.944700

Every time this happens, the terminal running softipbus-forward is updated with the message channel 3: open failed: connect failed:

~ # /tmp/softipbus-forward
channel 3: open failed: connect failed:
channel 3: open failed: connect failed:
channel 3: open failed: connect failed:

Just for fun, I tried doing this with softipbus running (as opposed to softipbus-forward) and it just segfaulted, but I never expected that to work anyway.

In case I'm screwing up something on the xmd side of things, here's my xmd input and output, left up in a separate window

nwoods@ayinger /afs/hep.wisc.edu/cms/nwoods  $ xmd
Xilinx Microprocessor Debugger (XMD) Engine
Xilinx EDK 14.4 Build EDK_P.49d
Copyright (c) 1995-2012 Xilinx, Inc.  All rights reserved.

XMD%
XMD% connect mb mdm -debugdevice deviceNr 2

JTAG chain configuration
--------------------------------------------------
Device   ID Code        IR Length    Part Name
 1       042a2093          10        XC6VHX250T
 2       442a8093          10        XC6VHX380T

MicroBlaze Processor Configuration :
-------------------------------------
Version............................8.40.b
Optimization.......................Performance
Interconnect.......................AXI-LE
MMU Type...........................No_MMU
No of PC Breakpoints...............2
No of Read Addr/Data Watchpoints...0
No of Write Addr/Data Watchpoints..0
Instruction Cache Support..........on
Instruction Cache Base Address.....0xf0000000
Instruction Cache High Address.....0xffffffff
Data Cache Support.................on
Data Cache Base Address............0xf0000000
Data Cache High Address............0xffffffff
Exceptions  Support................on
FPU  Support.......................off
Hard Divider Support...............off
Hard Multiplier Support............on - (Mul32)
Barrel Shifter Support.............off
MSR clr/set Instruction Support....on
Compare Instruction Support........on
Data Cache Write-back Support......off
Fault Tolerance Support............off
Stack Protection Support...........off

Connected to "mb" target. id = 0
Starting GDB server for "mb" target (id = 0) at TCP port no 1234
XMD% read_uart
MDM Uart Present in the System

Connected to MDM UART Target
Channel Opened

XMD% dow cms-calo-layer1/ctp6_fe_uart_ipbus/payload.elf
Downloading Program -- cms-calo-layer1/ctp6_fe_uart_ipbus/payload.elf
section, .vectors.reset: 0x00000000-0x00000007
section, .vectors.sw_exception: 0x00000008-0x0000000f
section, .vectors.interrupt: 0x00000010-0x00000017
section, .vectors.hw_exception: 0x00000020-0x00000027
section, .text: 0x00000050-0x0000d77f
section, .init: 0x0000d780-0x0000d7bb
section, .fini: 0x0000d7bc-0x0000d7db
section, .ctors: 0x0000d7dc-0x0000d7e3
section, .dtors: 0x0000d7e4-0x0000d7eb
section, .rodata: 0x0000d7ec-0x0000e5a7
section, .data: 0x0000e5a8-0x0000ec43
section, .eh_frame: 0x0000ec44-0x0000ecbf
section, .jcr: 0x0000ecc0-0x0000ecc3
section, .sdata: 0x0000ecc4-0x0000ecc7
section, .bss: 0x0000ecc8-0x0000ef1b
section, .heap: 0x0000ef1c-0x00010f1f
section, .stack: 0x00010f20-0x00012f1f
Download Progress..10.20.30.40.50.60.70.80.90.Done
Setting PC with Program Start Address 0x00000000
System Reset .... DONE

XMD% run
Processor started. Type "stop" to stop processor

RUNNING> 0
XMD%

[ekfriis]

Did the non-forward version segfault immediately, or when you tried to run the cli.py? If you

Can you send a PR to ctp6commander with your fixes?

One idea - set the LOG_LEVEL to be 3 when you compile softibpus-forward for the backend, then it should print some more info to the console. Also, if you add "-vvv" to your ssh command which sets up the forwarding (via the -L option), it should print some "connect open" debug message or somesuch when you try to connect over that port.

On Mon, Oct 14, 2013 at 9:21 PM, nwoods notifications@github.com wrote:

I fixed the biggest few bugs in ctp6commander cli.py and api.py, and it gets a lot further now than before. Now it fails at the line hw.dispatch() (where hw is ctp6.frontend) with the message

nwoods@ayinger /afs/hep.wisc.edu/cms/nwoods/ctp6commander master$ python cli.py --connection ctp6_connections.xml status 14-10-13 13:45:31.944377 [0x1f6898f0] WARNING - Closing Socket since exception detected. Traceback (most recent call last): File "cli.py", line 167, in commands[args.command](hw, args) File "cli.py", line 48, in do_status status_flags = api.status(hw, args.links) File "/afs/hep.wisc.edu/cms/nwoods/ctp6commander/api.py", line 156, in status hw.dispatch() uhal._core.exception: I'm terribly sorry to have to tell you this, but it appears that there was an exception:

• Exception type: uhal::exception::IPbus2PacketHeaderMismatch
• Description: Exception class to handle the case where the IPbus 2.0 packet header does not match that sent.
• Additional Information:

  Returned Packet Header 0x00000000 does not match that sent 0x200000F0

• Exception occured in thread with ID: 0x1f6a8bf0
• Current thread has ID: 0x1f6898f0
• Exception constructed at time: 2013-10-14 13:45:31.944187
• Exception's what() function called at time: 2013-10-14 13:45:31.944700

Every time this happens, the terminal running softipbus-forward is updated with the message channel 3: open failed: connect failed:

~ # /tmp/softipbus-forward channel 3: open failed: connect failed: channel 3: open failed: connect failed: channel 3: open failed: connect failed:

Just for fun, I tried doing this with softipbus running (as opposed to softipbus-forward) and it just segfaulted, but I never expected that to work anyway.

In case I'm screwing up something on the xmd side of things, here's my xmd input and output, left up in a separate window

nwoods@ayinger /afs/hep.wisc.edu/cms/nwoods $ xmd Xilinx Microprocessor Debugger (XMD) Engine Xilinx EDK 14.4 Build EDK_P.49d Copyright (c) 1995-2012 Xilinx, Inc. All rights reserved.

XMD% XMD% connect mb mdm -debugdevice deviceNr 2

JTAG chain configuration

Device ID Code IR Length Part Name 1 042a2093 10 XC6VHX250T 2 442a8093 10 XC6VHX380T

MicroBlaze Processor Configuration :

Version............................8.40.b Optimization.......................Performance Interconnect.......................AXI-LE MMU Type...........................No_MMU No of PC Breakpoints...............2 No of Read Addr/Data Watchpoints...0 No of Write Addr/Data Watchpoints..0 Instruction Cache Support..........on Instruction Cache Base Address.....0xf0000000 Instruction Cache High Address.....0xffffffff Data Cache Support.................on Data Cache Base Address............0xf0000000 Data Cache High Address............0xffffffff Exceptions Support................on FPU Support.......................off Hard Divider Support...............off Hard Multiplier Support............on - (Mul32) Barrel Shifter Support.............off MSR clr/set Instruction Support....on Compare Instruction Support........on Data Cache Write-back Support......off Fault Tolerance Support............off Stack Protection Support...........off

Connected to "mb" target. id = 0 Starting GDB server for "mb" target (id = 0) at TCP port no 1234 XMD% read_uart MDM Uart Present in the System

Connected to MDM UART Target Channel Opened

XMD% dow cms-calo-layer1/ctp6_fe_uart_ipbus/payload.elf Downloading Program -- cms-calo-layer1/ctp6_fe_uart_ipbus/payload.elf section, .vectors.reset: 0x00000000-0x00000007 section, .vectors.sw_exception: 0x00000008-0x0000000f section, .vectors.interrupt: 0x00000010-0x00000017 section, .vectors.hw_exception: 0x00000020-0x00000027 section, .text: 0x00000050-0x0000d77f section, .init: 0x0000d780-0x0000d7bb section, .fini: 0x0000d7bc-0x0000d7db section, .ctors: 0x0000d7dc-0x0000d7e3 section, .dtors: 0x0000d7e4-0x0000d7eb section, .rodata: 0x0000d7ec-0x0000e5a7 section, .data: 0x0000e5a8-0x0000ec43 section, .eh_frame: 0x0000ec44-0x0000ecbf section, .jcr: 0x0000ecc0-0x0000ecc3 section, .sdata: 0x0000ecc4-0x0000ecc7 section, .bss: 0x0000ecc8-0x0000ef1b section, .heap: 0x0000ef1c-0x00010f1f section, .stack: 0x00010f20-0x00012f1f Download Progress..10.20.30.40.50.60.70.80.90.Done Setting PC with Program Start Address 0x00000000 System Reset .... DONE

XMD% run Processor started. Type "stop" to stop processor

RUNNING> 0 XMD%

— Reply to this email directly or view it on GitHubhttps://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26281376 .

[nwoods]

Pull request made. I fixed a number of bugs and added the necessary xml files to the ctp6commander root directory (env/bin/activate will set $CTP6_CONNECTION if you run setup_venv.sh fresh. If you aren't planning to, just run the relevant 3 lines from it by hand once).

softipbus non-forward segfaulted only when cli.py was run.

I'll try your suggestions and see if I can at least get some more information.

[nwoods]

Setting the log level to 3 does not appear to change softipbus's output.

On Mon, Oct 14, 2013 at 4:41 PM, Nate Woods nwoods@hep.wisc.edu wrote:

Pull request made. I fixed a number of bugs and added the necessary xml files to the ctp6commander root directory (env/bin/activate will set $CTP6_CONNECTION if you run setup_venv.sh fresh. If you aren't planning to, just run the relevant 3 lines from it by hand once).

softipbus non-forward segfaulted only when cli.py was run.

I'll try your suggestions and see if I can at least get some more information. Did the non-forward version segfault immediately, or when you tried to run the cli.py? If you

Can you send a PR to ctp6commander with your fixes?

One idea - set the LOG_LEVEL to be 3 when you compile softibpus-forward for the backend, then it should print some more info to the console. Also, if you add "-vvv" to your ssh command which sets up the forwarding (via the -L option), it should print some "connect open" debug message or somesuch when you try to connect over that port.

On Mon, Oct 14, 2013 at 9:21 PM, nwoods notifications@github.com wrote:

I fixed the biggest few bugs in ctp6commander cli.py and api.py, and it gets a lot further now than before. Now it fails at the line hw.dispatch() (where hw is ctp6.frontend) with the message

nwoods@ayinger /afs/hep.wisc.edu/cms/nwoods/ctp6commander master$ python cli.py --connection ctp6_connections.xml status 14-10-13 13:45:31.944377 [0x1f6898f0] WARNING - Closing Socket since exception detected. Traceback (most recent call last): File "cli.py", line 167, in commands[args.command](hw, args) File "cli.py", line 48, in do_status status_flags = api.status(hw, args.links) File "/afs/hep.wisc.edu/cms/nwoods/ctp6commander/api.py", line 156, in status hw.dispatch() uhal._core.exception: I'm terribly sorry to have to tell you this, but it appears that there was an exception:

• Exception type: uhal::exception::IPbus2PacketHeaderMismatch
• Description: Exception class to handle the case where the IPbus 2.0 packet header does not match that sent.
• Additional Information:

  Returned Packet Header 0x00000000 does not match that sent 0x200000F0

• Exception occured in thread with ID: 0x1f6a8bf0
• Current thread has ID: 0x1f6898f0
• Exception constructed at time: 2013-10-14 13:45:31.944187
• Exception's what() function called at time: 2013-10-14 13:45:31.944700

Every time this happens, the terminal running softipbus-forward is updated with the message channel 3: open failed: connect failed:

~ # /tmp/softipbus-forward channel 3: open failed: connect failed: channel 3: open failed: connect failed: channel 3: open failed: connect failed:

Just for fun, I tried doing this with softipbus running (as opposed to softipbus-forward) and it just segfaulted, but I never expected that to work anyway.

In case I'm screwing up something on the xmd side of things, here's my xmd input and output, left up in a separate window

nwoods@ayinger /afs/hep.wisc.edu/cms/nwoods $ xmd Xilinx Microprocessor Debugger (XMD) Engine Xilinx EDK 14.4 Build EDK_P.49d Copyright (c) 1995-2012 Xilinx, Inc. All rights reserved.

XMD% XMD% connect mb mdm -debugdevice deviceNr 2

JTAG chain configuration

Device ID Code IR Length Part Name 1 042a2093 10 XC6VHX250T 2 442a8093 10 XC6VHX380T

MicroBlaze Processor Configuration :

Version............................8.40.b Optimization.......................Performance Interconnect.......................AXI-LE MMU Type...........................No_MMU No of PC Breakpoints...............2 No of Read Addr/Data Watchpoints...0 No of Write Addr/Data Watchpoints..0 Instruction Cache Support..........on Instruction Cache Base Address.....0xf0000000 Instruction Cache High Address.....0xffffffff Data Cache Support.................on Data Cache Base Address............0xf0000000 Data Cache High Address............0xffffffff Exceptions Support................on FPU Support.......................off Hard Divider Support...............off Hard Multiplier Support............on - (Mul32) Barrel Shifter Support.............off MSR clr/set Instruction Support....on Compare Instruction Support........on Data Cache Write-back Support......off Fault Tolerance Support............off Stack Protection Support...........off

Connected to "mb" target. id = 0 Starting GDB server for "mb" target (id = 0) at TCP port no 1234 XMD% read_uart MDM Uart Present in the System

Connected to MDM UART Target Channel Opened

XMD% dow cms-calo-layer1/ctp6_fe_uart_ipbus/payload.elf Downloading Program -- cms-calo-layer1/ctp6_fe_uart_ipbus/payload.elf section, .vectors.reset: 0x00000000-0x00000007 section, .vectors.sw_exception: 0x00000008-0x0000000f section, .vectors.interrupt: 0x00000010-0x00000017 section, .vectors.hw_exception: 0x00000020-0x00000027 section, .text: 0x00000050-0x0000d77f section, .init: 0x0000d780-0x0000d7bb section, .fini: 0x0000d7bc-0x0000d7db section, .ctors: 0x0000d7dc-0x0000d7e3 section, .dtors: 0x0000d7e4-0x0000d7eb section, .rodata: 0x0000d7ec-0x0000e5a7 section, .data: 0x0000e5a8-0x0000ec43 section, .eh_frame: 0x0000ec44-0x0000ecbf section, .jcr: 0x0000ecc0-0x0000ecc3 section, .sdata: 0x0000ecc4-0x0000ecc7 section, .bss: 0x0000ecc8-0x0000ef1b section, .heap: 0x0000ef1c-0x00010f1f section, .stack: 0x00010f20-0x00012f1f Download Progress..10.20.30.40.50.60.70.80.90.Done Setting PC with Program Start Address 0x00000000 System Reset .... DONE

XMD% run Processor started. Type "stop" to stop processor

RUNNING> 0 XMD%

— Reply to this email directly or view it on GitHub< https://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26281376> .

— Reply to this email directly or view it on GitHubhttps://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26281756 .

[nwoods]

Maybe the -vvv option on ssh gets us somewhere. When I did that, the error message became

~ # /tmp/softipbus-forward 
debug1: Connection to port 60002 forwarding to 0.0.0.0 port 60002 requested.
debug2: fd 9 setting TCP_NODELAY
debug2: fd 9 setting O_NONBLOCK
debug3: fd 9 is O_NONBLOCK
debug1: channel 3: new [direct-tcpip]
channel 3: open failed: connect failed: 
debug1: channel 3: free: direct-tcpip: listening port 60002 for 0.0.0.0 port 60002, connect from 127.0.0.1 port 46888, nchannels 4
debug3: channel 3: status: The following connections are open:
  #2 client-session (t4 r0 i0/0 o0/0 fd 6/7 cfd -1)
  #3 direct-tcpip: listening port 60002 for 0.0.0.0 port 60002, connect from 127.0.0.1 port 46888 (t3 r-1 i0/0 o0/0 fd 9/9 cfd -1)

debug3: channel 3: close_fds r 9 w 9 e -1 c -1

I don't know what that means, but I'm sure someone does...

I connected to the CTP with the command

ssh -vvv -L 60002:0.0.0.0:60002 root@192.168.1.31

[ekfriis]

Did you make clean and re-make after changing the LOG_LEVEL?

Also, can you check in the makefile tha 60002 is the correct port for softipbus-forward?

On Tue, Oct 15, 2013 at 12:04 AM, nwoods notifications@github.com wrote:

Maybe the -vvv option on ssh gets us somewhere. When I did that, the error message became

~ # /tmp/softipbus-forward debug1: Connection to port 60002 forwarding to 0.0.0.0 port 60002 requested. debug2: fd 9 setting TCP_NODELAY debug2: fd 9 setting O_NONBLOCK debug3: fd 9 is O_NONBLOCK debug1: channel 3: new [direct-tcpip] channel 3: open failed: connect failed: debug1: channel 3: free: direct-tcpip: listening port 60002 for 0.0.0.0 port 60002, connect from 127.0.0.1 port 46888, nchannels 4 debug3: channel 3: status: The following connections are open:

2 client-session (t4 r0 i0/0 o0/0 fd 6/7 cfd -1)

3 direct-tcpip: listening port 60002 for 0.0.0.0 port 60002, connect from 127.0.0.1 port 46888 (t3 r-1 i0/0 o0/0 fd 9/9 cfd -1)

debug3: channel 3: close_fds r 9 w 9 e -1 c -1

I don't know what that means, but I'm sure someone does...

I connected to the CTP with the command

ssh -vvv -L 60002:0.0.0.0:60002 root@192.168.1.31

— Reply to this email directly or view it on GitHubhttps://github.com/uwcms/cms-calo-layer1/issues/4#issuecomment-26292763 .

[ekfriis]

How to commit changes to the softipbus SVN:

cd ~/softipbus # or wherever
svn status  # shows what you have modified
svn update # make sure you are on the head
svn commit -m "a helpful message" file_to_commit another_file yet_another_file

[nwoods]

A few questions on Mathias's behalf:

Can you explain the difference between softipbus and softipbus-forward? I've used softipbus-forward, and I've been scp'ing softipbus to the card every time I recompile it, but I don't actually know when I'm supposed to use it. From the names, it sounds like both need to be running for things to work, but that's not possible.

Also, if we want to include these in the petalinux flash, how do we do that?

[ekfriis]

explain the difference between softipbus and softipbus-forward

softipbus and softipbus-forward are separate programs.

softipbus runs a TCP server which reads out the local memory - so this is what needs to run to read memory from the backend, if we want to. We have not needed this as a use-case to date. Sidebar: once we get the ZYNQ chip, we will only need this program, since the Mathias will do magic to make the front end appear as back-end memory.

softipbus-forward runs a TCP server (on the backend), and then forwards the packets across the UART, to the front end. The UART devices forwarded over are defined in the softipbus makefile [1] - the current defaults should be correct.

Note that you also need to run the standalone program (ctp6_fe_uart_ipbus) on the front end [2], which reads the data sent on the UART, parses it, and sends a response back along the UART. The backend then sends this back over TCP.

if we want to include these in the petalinux flash, how do we do that?

ask Jes. at some point she added it, but I'm not sure what happened to that.

[1] https://svnweb.cern.ch/trac/cactus/browser/trunk/cactuscore/softipbus/Makefile#L35 [2] https://github.com/uwcms/cms-calo-layer1/tree/master/ctp6_fe_uart_ipbus