Add support for the PS4 camera

HipsterSloth commented 8 years ago

The ps4 camera has two HD RGB cameras a fixed distance apart:

things-you ll-be-able-to-do-with-the-new-ps4-camera

This is nice for several reasons:

The higher resolution gives you farther tracking range (like 8-9 feet instead of 5-6 feet)
You don't need to use a calibration mat or hmd coregistration to figure out the the camera poses relative to each other (since they are a known fixed distance apart)
You can use the two camera to triangulate the z-position of the controller more accurately then with just one camera

The one downside is that Sony decided to use a non-standard plug for the camera even though it's an ordinary USB3 device. Fortunately a kind soul make a circuit board pattern you can order that lets you put a PS4AUX connector on one end and a USB3 plug on the other:

https://oshpark.com/shared_projects/0PjWmdWL

The PS4AUX plug can be bought from AliExpress (takes a while since it ships from China):

http://www.aliexpress.com/snapshot/7705224378.html?orderId=75474520179370

~~And the USB3 plugin can be ordered from Amazon:~~

~~http://www.amazon.com/Degree-Panel-USB3-0-Female-Socket/dp/B00YX8UDGQ~~ NOTE: You can order this USB connector, but you have to bend the pins. Probably better to order the Digikey connector that the board was designed for.

And the USB3 plugin can be ordered from Digikey:

http://www.digikey.com/product-detail/en/692112030100/732-3157-ND/2627273

Finally there is a PS4EYECam project up on git hub we can use for the actual camera driver: https://github.com/bigboss-ps3dev/PS4EYECam

TianKiat commented 8 years ago

When can the support for this camera be implemented, I am dying to try this service out but I only have this camera instead of the PS3 eye. Thanks

HipsterSloth commented 8 years ago

It's one of the two major features I'm planning on working on next. Getting this working and getting the DualShock4 working. I need both of those for work. Just make sure that you have ordered the parts above for the adapter. The PS4Aux adapter takes a while because it has to ship from China.

demonixis commented 8 years ago

The adaptater is only used if you want to use the camera with a PS4 right? Do you have a schema for replacing the PS4Aux adapter by a USB adapter?

HipsterSloth commented 8 years ago

@demonixis The adapter posted above is so that you can connect the camera to your PC via usb. The camera has a ps4aux connector on it's cable, not a usb connector.

demonixis commented 8 years ago

@HipsterSloth Yes I know but the PCB have no components, it's basically a remapping of wires from the PS4 connector to a USB connector right? So it must be possible to cut the PS4Aux socket and solder wires to an USB connector. I don't have the PS4 and I can buy a PS4Eye Camera and hack it ;) Can you confirme me that it can works?

HipsterSloth commented 8 years ago

@demonixis Oh sure you can just manually splice the wires to a usb cable if you want. There was a page here showing the wire remapping:

http://www.psdevwiki.com/ps4/Aux

And someone from this thread soldering a usb cable to the ps4 aux cable:

https://02659908020089878153.googlegroups.com/attach/a30a8042608e1ed/IMG_20150115_205430%5B1%5D.jpg?part=0.1&view=1&vt=ANaJVrFUC2DFhuFWjfaEx0od-WWuXe7NEav6M8iWj62YXRHUsU7LKAEI0sHYUV_XYSR414ta1p4zENvW0s5geN2G1Vu6ZlJdj82ju2V8IuMnCxhwxHkFN9Q

Can you confirme me that it can works?

I'm not going to cut into my PS4 camera cable because I going with the adapter route. I was going to solder the adapter this week and try it out. As soon as I have tested and confirmed the adapter works I'll let you know.

demonixis commented 8 years ago

That's great! The PS4 Camera is not too expensive, if the tracking is better with this camera, I'll buy it for sure. What is the work frequency of the camera? Kinect works at 30Hz, is it the same for the PS4 Camera?

HipsterSloth commented 8 years ago

The two cameras can be set to run at the following resolutions/frame rates:

1280x800 pixel @ 60fps 640x400 pixel @ 120fps 320x192 pixel @ 240fps

demonixis commented 8 years ago

Hum the tracking must be great @ 120hz, I read on the internet that PS Move are bad with PSVR because of tracking quality. Do you think that it can be improved by using two cameras or is it an internal problem?

HipsterSloth commented 8 years ago

I actually tried a demo of the PSVR today at a BestBuy. Tracking quality was surprisingly good for the PSMove. I think they used 1280x800 pixel @ 60fps by default. The combination of of the higher video frame resolution and having two cameras in the PS4 camera to triangulate seems to eliminate most noticeable z-jitter that you get from a single PS3Eye camera. That said, because it's a regular RGB camera being used to filter a tracking color (as compared to a flashing IR signal), it's going to be subject to more local lighting condition issues then an Oculus (or OSVR) IR tracking camera.

demonixis commented 8 years ago

Sounds good! I'll get this camera when the ps4camera branch will be merged

psxdev commented 8 years ago

hello there, cool project indeed. I could help with ps4 camera integration.

I have here a few ps4 cameras, ps move, gear vr and oculus dk2 waiting me to make things happens , and your project can be very interesting for all people looking for cheap vr/ar open source controllers based on ps4 camera + ps move on pc/mac.

HipsterSloth commented 8 years ago

@psxdev I was actually planning on contacting you about this! Thanks a bunch for reaching out. And thanks for your work on PS4EyeCam.

At home I'm been trying to get PS4EyeCam working on my Win10 machine. I've built the PS4AUX-to-USB3 adapter mentioned in the links above and made an SDL wrapper test app (only because I was too lazy to get Cinder working).

I built the custom version of libusb recommended on stbnps fork that folds in the winiso support (since the windows version libusb doesn't support isosynchronous tranfers by default).

With this I was able to get the firmware uploading and onto the camera. Once the firmware was loaded it looked like I could get past the camera initialization sequence, but the the iso transfers appeared to keep timing out on me and thus video stream doesn't start.

Is there some libusb logging I can turn on that would make it clear why the iso transfers were failing?

psxdev commented 8 years ago

Try to set libusb_set_debug to 4. [http://libusb.sourceforge.net/api-1.0/group__lib.html]

I have not w10 hardware available for testing i didn't get run it fine on vbox w10 machine because my main platform is osx. I have report success from @stbnps only so windows is not user friendly for PS4EyeCam right now.

Did you test it on osx or linux or directly with ps4 to confirm that your ps4 camera device is fine?

HipsterSloth commented 8 years ago

@psxdev I will give the libusb_set_debug setting a try tonight and post the logs here. I'll also put my test code with cmake files so that others can replicate my test case.

Good question about testing the camera. I haven't yet tried on OS X or Linux. I have an older MacMini I can test on. I need to check if it has USB3 ports or not. If it doesn't I can run Linux off a thumbdrive on my desktop PC. I have tested the camera on a PS4 and verified that at least the camera is working (and my adapter didn't toast the camera).

In any case, it would still be good for me to verify the camera code runs on a non-windows machine just to define a known working baseline.

HipsterSloth commented 8 years ago

@psxdev I was able to get a high detail log file of libusb while trying to run the camera in Win10 (see below). Here'e what I did to get there:

1) I made a fork of the most recent version of libusb and added the following cherry picked changes from JoshBlake's libusbx fork

Adding isosynchronous transfer support via libusbk - 53e208c138883d0335d4422d61db024225ed2df4
Adding super speed iso synchronous length handling - c20fdf36968b3b3891ef95dc11eaedfdefd8d05a

2) I made a fork of PS4EyeCam with the following changes to make it build

A few simple Win32 compile error fixes: 56148a6641eb8aa7cd65a232ae77b5d7d6cb7cf9
Added SDL2 and my fork of libusb as submodules - cced8695901833138d7f35112af85c41ec036b2a
Added a simple SDL camera test app that mirrors what the other two test apps do along with cmake scripts and batch files to build everything - bdccecdea5af32addf73ee10e03f51d1549fda2f

3) Connected my ps4 camera to my pc and made sure it just showed the USB Boot entry:

prefirmware

4) Installed lib-usb onto the USB boot device

usbbootinstall

5) Launced SDLTest, my simple SDL test app built using PS4EyeCam, which installs the firmware on the camera. Windows then appears to install some new drivers for a camera in response.

postfirmware

7) Select the OV580 composite parent entry in Zadig and install libusbK

ov580_libusbk

6) Launch SDLTest again which connects to the camera and attempts to setup mode 1 and stream camera data. This generates the following log:

log.zip

The most interesting bit is this part:

 [ 3.489788] [000029a4] libusb: debug [libusb_submit_transfer] transfer 00AF4590
[ 3.489788] [000029a4] libusb: debug [winusbx_submit_iso_transfer] matched endpoint 81 with interface 1
[ 3.490791] [000029a4] libusb: debug [winusbx_submit_iso_transfer] reading 8 iso packets
[ 3.490791] [000029a4] libusb: debug [usbi_add_pollfd] add fd 1 events 1
[ 3.490791] [000029a4] libusb: debug [windows_handle_events] checking fd 2 with revents = 0001
[ 3.490791] [000029a4] libusb: debug [usbi_remove_pollfd] remove fd 2
[ 3.490791] [000029a4] libusb: debug [windows_transfer_callback] handling I/O completion with errcode 1784, size 0
[ 3.490791] [000029a4] libusb: error [windows_transfer_callback] detected I/O error 1784: [1784] The supplied user buffer is not valid for the requested operation.

That I/O error 1784 is getting hit non-stop. I think that 's my problem. But I don't know why the transfer buffer is invalid. I'm wondering if there is something not right with the "super speed iso synchronous length handling" patch - c20fdf36968b3b3891ef95dc11eaedfdefd8d05a.

If you want to try building my version of PS4EyeCam yourself do the following: 1) git clone --recursive https://github.com/HipsterSloth/PS4EYECam.git 2) Run InitialSetup_Win32.bat 3) Go to the newly generated /build folder and open PS3EyeCam.sln 4) Build the solution 5) Build the 'INSTALL' project 6) Go to the Win32/bin folder and run SDLTest.exe

Sadly my ancient macmini only has USB 2.0.

Tomorrow I'm going to try and build this in Linux on my same PC using the Ubuntu thumbdrive I built this evening. I'll post an update and let you know how that goes.

psxdev commented 8 years ago

I compiled your code on w10 vbox machine. I had changed some to adapt to my visual studio 14 and avoid call to SetBuildVars_x64.bat because is not initialized in my system.

I attach camera previously initialized in osx so i am avoiding load firmware again on windows

Enable debug to check initialization check in ps4eye.h to compare with my initialization in osx working code https://github.com/HipsterSloth/PS4EYECam/blob/master/driver/include/ps4eye.h#L66

i disabled sanity check for sensor and mirror and now initilization is done and led is powered up

i got an sdl error in renderer and i changed to SDL_RENDERER_SOFTWARE to avoid it

after that i got payload error or error 87 it's a problem with virtual box i suppose i can't get fix it in virtual machine. Same problem described at https://github.com/bigboss-ps3dev/PS4EYECam/issues/1

output from osx working initialization code: Found PlayStation Camera Firmware already loaded... Found PlayStation Camera ERROR(AntTweakBar) >> Parsing error in def string: can't read attribute value [key ...] Device Descriptor: bLength 18 bDescriptorType 1 bcdUSB 3.00 bDeviceClass 239 bDeviceSubClass 2 bDeviceProtocol 1 bMaxPacketSize0 9 idVendor 0x05a9 idProduct 0x058a bcdDevice 1.00 iManufacturer 1 iProduct 2 iSerial 0 bNumConfigurations 1 Configuration Descriptor: bLength 9 bDescriptorType 2 wTotalLength 311 bNumInterfaces 2 bConfigurationValue 1 iConfiguration 0 bmAttributes 0x80 (Bus Powered) MaxPower 128mA Interface Association: bLength 8 bDescriptorType 11 bFirstInterface 0 bInterfaceCount 2 bFunctionClass 14 bFunctionSubClass 3 bFunctionProtocol 0 iFunction 2 Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 0 bAlternateSetting 0 bNumEndpoints 0 bInterfaceClass 14 bInterfaceSubClass 1 bInterfaceProtocol 0 iInterface 2 Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 1 bAlternateSetting 0 bNumEndpoints 0 bInterfaceClass 14 bInterfaceSubClass 2 bInterfaceProtocol 0 iInterface 0 Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 1 bAlternateSetting 1 bNumEndpoints 1 bInterfaceClass 14 bInterfaceSubClass 2 bInterfaceProtocol 0 iInterface 0 Endpoint: bEndpointAddress: 81h bmAttributes: 05h wMaxPacketSize: 1024 bInterval: 1 bRefresh: 0 bSynchAddress: 0 USB 3.0 Endpoint Companion: bMaxBurst: 15 bmAttributes: 0x02 wBytesPerInterval: 49152 configuration set to 1 claim interface 1 set alt to 1 in interface 1 control transfer for video get return 26 bmHint: 0001 bFormatIndex: 0 bFrameIndex: 3 dwFrameInterval: 333333 wKeyFrameRate: 0 wPFrameRate: 0 wCompQuality: 0 wCompWindowSize: 0 wDelay: 8315 dwMaxVideoFrameSize: 359200 dwMaxPayloadTransferSize: 4294950912 bInterfaceNumber: 0 Frame resolutiion: 640x400 framerate: 30 control transfer for video set return 26 control transfer for video get return 26 bmHint: 0001 bFormatIndex: 1 bFrameIndex: 2 dwFrameInterval: 333333 wKeyFrameRate: 0 wPFrameRate: 0 wCompQuality: 0 wCompWindowSize: 0 wDelay: 49347 dwMaxVideoFrameSize: 1426368 dwMaxPayloadTransferSize: 49152 bInterfaceNumber: 0 read sensor_id from sensor 1 sensor 1 id 0xffff Sensor 1 is not ready, exit Error reading Sensor 1 Sanity check from sensors failed frame number 1 i=0 Write register 0xff70 to f1 BEGIN MULTIWRITE USB brequest a4

USB DATA HEADER BYTE 8 TO 15 0x03 0x00 0x00 0x00 0x00 0x00 0x00 0x00 USB DATA REGISTER LIST 16 0x70 0xff 0xf1 0xff END MULTIWRITE USB frame number 2 i=1 Write register 0xff70 to 01 BEGIN MULTIWRITE USB brequest a4

USB DATA HEADER BYTE 8 TO 15 0x03 0x00 0x00 0x00 0x00 0x00 0x00 0x00 USB DATA REGISTER LIST 16 0x71 0xff 0x01 0xff END MULTIWRITE USB i=1 val=1 frame number 3 i=2 val=2 frame number 4 Control transfer error: Stall phase 2 read failed received 0 bytes instead of 64 phase 2 read failed register read 0x0000 not 0xff71 i=3 val=0 i=3 val=3 frame number 5 4 val=4 frame number 6 5 val=0 frame number 7 frame number 8 frame number 9 frame number 10 frame number 11 frame number 12 frame number 13 frame number 14 frame number 15 frame number 16 .... ... ... frame number 370 Shutdown begin wait... PS4EYECAM shutdown called wait... stop is called is_streaming is: 1 set_led_off called libusb: warning [darwin_abort_transfers] aborting all transactions on interface 1 pipe 1 libusb: warning [darwin_abort_transfers] aborting all transactions on interface 1 pipe 1 libusb: warning [darwin_abort_transfers] aborting all transactions on interface 1 pipe 1 libusb: warning [darwin_abort_transfers] aborting all transactions on interface 1 pipe 1 libusb: warning [darwin_abort_transfers] aborting all transactions on interface 1 pipe 1 libusb: warning [darwin_abort_transfers] aborting all transactions on interface 1 pipe 1 libusb: warning [darwin_abort_transfers] aborting all transactions on interface 1 pipe 1 libusb: warning [darwin_abort_transfers] aborting all transactions on interface 1 pipe 1 PS4EYECAM destructor stop is called is_streaming is: 0 PS4EYECAM release called PS4EYECAM destructor stop is called is_streaming is: 0 PS4EYECAM release called closing device device closed URBDesc destructor USBMgr destructor called

i attach debug output from osx debugosx.txt

HipsterSloth commented 8 years ago

@psxdev Thanks for the log and the compile attempt in windows! I'll turn on debugging tonight and give this a try. both in windows and hopefully linux.

HipsterSloth commented 8 years ago

@psxdev Had to fix a few missing time related functions and some issues with my cmake files, but I finally got it compiling under Linux. I used the same build from github fork mentioned before.

It's currently failing on the call to libusb_set_configuration. I ran out of time to dig into that tonight. Here's the log from my SDLTest in case you are interested:

ubuntu@ubuntu:~/PS4EYECam/build/sdl$ sudo ./SDLTest

[timestamp] [threadID] facility level [function call]

[ 0.011663] [0000438a] libusb: debug [libusb_get_device_list] [ 0.011708] [0000438a] libusb: debug [discovered_devs_append] need to increase capacity [ 0.011714] [0000438a] libusb: debug [discovered_devs_append] need to increase capacity [ 0.011718] [0000438a] libusb: debug [discovered_devs_append] need to increase capacity [ 0.011725] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011728] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011730] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011733] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011735] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011738] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011740] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011745] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011750] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011752] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011755] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011757] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011760] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011762] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011765] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011767] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011770] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011772] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011775] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011777] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011779] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011782] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011784] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011787] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011789] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011792] [0000438a] libusb: debug [libusb_get_device_descriptor] Found PlayStation Camera [ 0.011857] [0000438a] libusb: debug [libusb_alloc_transfer] transfer 0x180a978 [ 0.011868] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011874] [0000438a] libusb: debug [libusb_get_device_descriptor] [ 0.011879] [0000438a] libusb: debug [libusb_get_device_descriptor] Firmware already loaded... [ 0.018085] [0000438a] libusb: debug [libusb_open] open 6.2 [ 0.100177] [0000438a] libusb: debug [usbi_add_pollfd] add fd 9 events 4 [ 0.100201] [0000438a] libusb: debug [libusb_get_device_descriptor] Device Descriptor: bLength 18 bDescriptorType 1 bcdUSB 3.00 bDeviceClass 239 bDeviceSubClass 2 bDeviceProtocol 1 bMaxPacketSize0 9 idVendor 0x05a9 idProduct 0x058a bcdDevice 1.00 iManufacturer 1 iProduct 2 iSerial 0 bNumConfigurations 1 [ 0.100225] [0000438a] libusb: debug [libusb_get_config_descriptor] index 0 [ 0.100235] [0000438a] libusb: debug [parse_configuration] skipping descriptor 0xb [ 0.100241] [0000438a] libusb: debug [parse_endpoint] skipping descriptor 30 Configuration Descriptor: bLength 9 bDescriptorType 2 wTotalLength 311 bNumInterfaces 2 bConfigurationValue 1 iConfiguration 0 bmAttributes 0x80 (Bus Powered) MaxPower 128mA Interface Association: bLength 8 bDescriptorType 11 bFirstInterface 0 bInterfaceCount 2 bFunctionClass 14 bFunctionSubClass 3 bFunctionProtocol 0 iFunction 2 Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 0 bAlternateSetting 0 bNumEndpoints 0 bInterfaceClass 14 bInterfaceSubClass 1 bInterfaceProtocol 0 iInterface 2 Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 1 bAlternateSetting 0 bNumEndpoints 0 bInterfaceClass 14 bInterfaceSubClass 2 bInterfaceProtocol 0 iInterface 0 Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 1 bAlternateSetting 1 bNumEndpoints 1 bInterfaceClass 14 bInterfaceSubClass 2 bInterfaceProtocol 0 iInterface 0 Endpoint: bEndpointAddress: 81h bmAttributes: 05h wMaxPacketSize: 1024 bInterval: 1 bRefresh: 0 bSynchAddress: 0 USB 3.0 Endpoint Companion: bMaxBurst: 15 bmAttributes: 0x02 wBytesPerInterval: 49152 [ 0.100319] [0000438a] libusb: debug [libusb_set_configuration] configuration 1 set configuration error: -6 PS3 Eye camera(0) failed to initialize

psxdev commented 8 years ago

You have other driver managing ps4 camera in your system. Check if uvc kernel module is claiming the interface. You can disable the uvc module or call libusb_detach_kernel_driver with the correct interface before claiming interface and it should fix that error -6 (LIBUSB_ERROR_BUSY)

HipsterSloth commented 8 years ago

@psxdev Success!

test

Looks like the kernel was in fact using the device:

ubuntu@ubuntu:~/PS4EYECam/build/sdl$ dmesg | grep "video"
[   12.864594] Linux video capture interface: v2.00
[   12.885480] uvcvideo: Found UVC 1.00 device USB Camera-OV580 (05a9:058a)
[   12.885969] uvcvideo: Found UVC 1.00 device MMP SDK (0bb4:2c87)
[   12.913711] usbcore: registered new interface driver uvcvideo

I tried calling libusb_detach_kernel_driver for interface 1, but I was still getting the same error, so I went ahead and just disabled the module in question:

ubuntu@ubuntu:~/PS4EYECam/build/sdl$ sudo modprobe -r uvcvideo
ubuntu@ubuntu:~/PS4EYECam/build/sdl$ lsmod | grep uvc
ubuntu@ubuntu:~/PS4EYECam/build/sdl$

With that the video stream started but was pretty garbled. Putting the call to ps4eye::PS4EYECam::updateDevices() in it's own thread seemed to fix that.

I checked all of this work into my fork in case you were curious: 90f6ec7d797aeac43b2b5592c35f859f6f71ea99

Now that I have a working base case I can compare the debug output when running in windows and see if I can determine where it's breaking down.

Here is the debug log for this video stream log.txt.gz

psxdev commented 8 years ago

Cool. Congratulations. I am glad to see people using my code. I have full reversed Sony driver from 1.76 ps4 console kernel and i am working in a new firmware and driver code with same features than original. I need free time to finish it.

I was compiling yesterday PSMoveService on OS X. I am using macport instead of brew stuff because all my playstation tools are using it. Perhaps i will make you some questions , i have generated project files and today i will try to compile it.

HipsterSloth commented 8 years ago

@psxdev Reverse engineering the camera firmware is an impressive feat. Were you able to get a dataset for the camera hardware? I recall you mentioning the data sheet was hard to come by because the manufacturer didn't make them publicly available.

And I'm glad to hear we have another mac user testing psmove service! @cboulay is the only other dev that builds on a mac regularly. I have my mac mini, but I can't test with my dk2 on it. Cboulay is the best person to ask about mac specific build issues, but we're both happy to answer questions.

psxdev commented 8 years ago

no datasheet for ov580 :( without signing NDA. There are a few companies using it (project tango, mantis vision, leapmotion, leopard imaging..) but their kits are expensive, sony is selling ps4 camera veeeeeery cheap.

firmware is embed in playstation 4 kernel in symbol called luke_rom_image (in 1.76 at address 0xffffffff82a58e30). I reversed low level fuctions from the ps4 kernel so now i know better how the driver works internally.

There are 4 different kinds of initial settings implemented for different cmos sensors/lens so it should be different hardware revisions. It is a slow job but it is very fun to discover new things about this device.

we can't extract camera firmware without a beagle usb 3 sniffer for ps4 console firmware newest than 1.76 so by now 1.76 is my base for experimental code.

First question i don't know it you like better to open a new issue for this:

@cboulay and @HipsterSloth i tried to generate projectfiles on osx but it doesn't generate well path for some libraries.

I followed these steps:

I installed initial dependencies with: sudo port install sdl sudo port install boost sudo port install protobuf-cpp

i had previously installed libusb and opencv

then i generate the projects file and open with xcode. When i try to compile it is searching some boost libraries but they are not in the system:

clang error: No such file or directory: '/opt/local/lib/libboost_date_time-mt.a'
clang: error: no such file or directory: '/opt/local/lib/libboost_regex-mt.a'

i have this from macport in /opt/local/lib there are the .dylib but i have not .a:

-rwxr-xr-x  1 root  admin     26664 24 nov  2015 libboost_atomic-mt.dylib
-rw-r--r--  1 root  admin    712408 24 nov  2015 libboost_chrono-mt.a
-rwxr-xr-x  1 root  admin    101268 24 nov  2015 libboost_chrono-mt.dylib
-rwxr-xr-x  1 root  admin    126192 24 nov  2015 libboost_container-mt.dylib
-rwxr-xr-x  1 root  admin    112072 24 nov  2015 libboost_context-mt.dylib
-rwxr-xr-x  1 root  admin    125580 24 nov  2015 libboost_coroutine-mt.dylib
-rwxr-xr-x  1 root  admin    159816 24 nov  2015 libboost_date_time-mt.dylib
-rw-r--r--  1 root  admin      6496 24 nov  2015 libboost_exception-mt.a
-rwxr-xr-x  1 root  admin    251220 24 nov  2015 libboost_filesystem-mt.dylib
-rwxr-xr-x  1 root  admin    973984 24 nov  2015 libboost_graph-mt.dylib
-rwxr-xr-x  1 root  admin    312064 24 nov  2015 libboost_iostreams-mt.dylib
-rwxr-xr-x  1 root  admin   2146732 24 nov  2015 libboost_locale-mt.dylib
-rwxr-xr-x  1 root  admin   2386340 24 nov  2015 libboost_log-mt.dylib
-rwxr-xr-x  1 root  admin   3241664 24 nov  2015 libboost_log_setup-mt.dylib
-rwxr-xr-x  1 root  admin    362008 24 nov  2015 libboost_math_c99-mt.dylib
-rwxr-xr-x  1 root  admin    224120 24 nov  2015 libboost_math_c99f-mt.dylib
-rwxr-xr-x  1 root  admin    243372 24 nov  2015 libboost_math_c99l-mt.dylib
-rwxr-xr-x  1 root  admin   1411580 24 nov  2015 libboost_math_tr1-mt.dylib
-rwxr-xr-x  1 root  admin   1388928 24 nov  2015 libboost_math_tr1f-mt.dylib
-rwxr-xr-x  1 root  admin   1356384 24 nov  2015 libboost_math_tr1l-mt.dylib
-rwxr-xr-x  1 root  admin    145528 24 nov  2015 libboost_prg_exec_monitor-mt.dylib
-rwxr-xr-x  1 root  admin   1195620 24 nov  2015 libboost_program_options-mt.dylib
-rwxr-xr-x  1 root  admin    886668 24 nov  2015 libboost_python-mt.dylib
-rwxr-xr-x  1 root  admin     83952 24 nov  2015 libboost_random-mt.dylib
-rwxr-xr-x  1 root  admin   2674544 24 nov  2015 libboost_regex-mt.dylib
-rwxr-xr-x  1 root  admin   1151428 24 nov  2015 libboost_serialization-mt.dylib
-rwxr-xr-x  1 root  admin    241096 24 nov  2015 libboost_signals-mt.dylib
-rw-r--r--  1 root  admin    139016 24 nov  2015 libboost_system-mt.a
-rwxr-xr-x  1 root  admin     58284 24 nov  2015 libboost_system-mt.dylib
-rw-r--r--  1 root  admin  20950288 24 nov  2015 libboost_test_exec_monitor-mt.a
-rwxr-xr-x  1 root  admin    301836 24 nov  2015 libboost_thread-mt.dylib
-rw-r--r--  1 root  admin    457456 24 nov  2015 libboost_timer-mt.a
-rwxr-xr-x  1 root  admin    106440 24 nov  2015 libboost_timer-mt.dylib
-rwxr-xr-x  1 root  admin   1922320 24 nov  2015 libboost_unit_test_framework-mt.dylib
-rwxr-xr-x  1 root  admin   4623804 24 nov  2015 libboost_wave-mt.dylib
-rwxr-xr-x  1 root  admin    844188 24 nov  2015 libboost_wserialization-mt.dylib

is there an updated document for compiling under osx and macport instead homebrew?

cboulay commented 8 years ago

What version of boost did you have? When installing with macports, is there any option to specify whether or not static libs should be installed?

Also, what version of protobuf did it install? We need >= 3.0 which is not installed by default with homebrew unless the --devel flag is passed.

Anyway, I'm not even sure we need those 2 boost modules that you're missing. I'll try to trim them out of the CMakeLists now and see what happens...

cboulay commented 8 years ago

OK they weren't needed. You should be able to get past that step. Edit: I just pushed the necessary change to the cmake file.

cboulay commented 8 years ago

Actually that's not the only target that had date-time and regex. PSMoveClient also had those modules in their list. I just removed them and was able to build PSMoveClient, but I also noticed a commit in the CAPI branch where @HipsterSloth explicitly added date-time to the config-tool: 4a4b37acaa758c08e3afe7f1ebb144116857cb25

So, even if the current master doesn't need regex or date-time, it will as soon as we merge the CAPI branch. Let's get rid of that boost::date-time and instead use std::chrono

(just pushed commit that fixes psmoveclient boost modules)

psxdev commented 8 years ago

build successfull.

custom build port for boost is needed. By default macport boost 1.59 is using shared only. I changed the port and build statically. so briefly:

sudo port install boost

modify port config file to build statically. It is at /opt/local/var/macports/sources/rsync.macports.org/release/tarballs/ports/devel/boost/Portfile.

sudo port build boost

after build copy static libraries generated at /opt/local/var/macports/sources/rsync.macports.org/release/tarballs/ports/devel/boost/work/boost_1_59_0/stage/lib to /opt/local/lib

-rw-r--r--  1 macports  admin     23360 14 jul 21:02 libboost_atomic-mt.a
-rw-r--r--  1 macports  admin    756576 14 jul 21:02 libboost_chrono-mt.a
-rw-r--r--  1 macports  admin    467376 14 jul 21:02 libboost_container-mt.a
-rw-r--r--  1 macports  admin    340768 14 jul 21:03 libboost_context-mt.a
-rw-r--r--  1 macports  admin    442136 14 jul 21:03 libboost_coroutine-mt.a
-rw-r--r--  1 macports  admin    881544 14 jul 21:03 libboost_date_time-mt.a
-rw-r--r--  1 macports  admin      6352 14 jul 21:02 libboost_exception-mt.a
-rw-r--r--  1 macports  admin   1874872 14 jul 21:03 libboost_filesystem-mt.a
-rw-r--r--  1 macports  admin   8736944 14 jul 21:03 libboost_graph-mt.a
-rw-r--r--  1 macports  admin   1637408 14 jul 21:03 libboost_iostreams-mt.a
-rw-r--r--  1 macports  admin  20891256 14 jul 21:03 libboost_locale-mt.a
-rw-r--r--  1 macports  admin  26710536 14 jul 21:04 libboost_log-mt.a
-rw-r--r--  1 macports  admin  37023984 14 jul 21:04 libboost_log_setup-mt.a
-rw-r--r--  1 macports  admin   1676208 14 jul 21:05 libboost_math_c99-mt.a
-rw-r--r--  1 macports  admin   1398640 14 jul 21:05 libboost_math_c99f-mt.a
-rw-r--r--  1 macports  admin   1460464 14 jul 21:05 libboost_math_c99l-mt.a
-rw-r--r--  1 macports  admin  26545584 14 jul 21:04 libboost_math_tr1-mt.a
-rw-r--r--  1 macports  admin  26613344 14 jul 21:04 libboost_math_tr1f-mt.a
-rw-r--r--  1 macports  admin  26310232 14 jul 21:05 libboost_math_tr1l-mt.a
-rw-r--r--  1 macports  admin    799176 14 jul 21:06 libboost_prg_exec_monitor-mt.a
-rw-r--r--  1 macports  admin  12369080 14 jul 21:05 libboost_program_options-mt.a
-rw-r--r--  1 macports  admin   9963080 14 jul 21:06 libboost_python-mt.a
-rw-r--r--  1 macports  admin    223288 14 jul 21:05 libboost_random-mt.a
-rw-r--r--  1 macports  admin  23037864 14 jul 21:03 libboost_regex-mt.a
-rw-r--r--  1 macports  admin  12059816 14 jul 21:06 libboost_serialization-mt.a
-rw-r--r--  1 macports  admin   2569128 14 jul 21:06 libboost_signals-mt.a
-rw-r--r--  1 macports  admin    138088 14 jul 21:02 libboost_system-mt.a
-rw-r--r--  1 macports  admin  21542960 14 jul 21:06 libboost_test_exec_monitor-mt.a
-rw-r--r--  1 macports  admin   1723080 14 jul 21:03 libboost_thread-mt.a
-rw-r--r--  1 macports  admin    457328 14 jul 21:06 libboost_timer-mt.a
-rw-r--r--  1 macports  admin  20916024 14 jul 21:06 libboost_unit_test_framework-mt.a
-rw-r--r--  1 macports  admin  53883080 14 jul 21:06 libboost_wave-mt.a
-rw-r--r--  1 macports  admin   8230144 14 jul 21:06 libboost_wserialization-mt.a

Then install protobuf 3.x

sudo port install protobuf3-cpp

generate project files and build

Thank you for your help

HipsterSloth commented 8 years ago

I just noticed that the "Removed date-time and regex from list of required boost modules." change (99d73f176a9e6368a8e3e6589079ce8db3f3eecc) in master broke the build in Windows build.

PSMoveService needed date_time for the boost based logging code. I just did a refactor to replace the boost based server logging code with same style logging we do on the client (b3c39228c22e81a937d6debab302aad7bcd79cde). This has the unfortunate side effect of not logging to a file anymore (it only logs to the console), but since we're not really running the PSMoveService as a real service yet, this doesn't really matter at the moment. I also want to do a proper refactor for the logging at some point anyway.

Boost::ASIO was also unnecessarily pulling in date_time and regex. I found this note in the boost.asio docs:

With MSVC or Borland C++ you may want to add -DBOOST_DATE_TIME_NO_LIB and -DBOOST_REGEX_NO_LIB to your project settings to disable autolinking of the Boost.Date_Time and Boost.Regex libraries respectively. Alternatively, you may choose to build these libraries and link to them.

I updated the cmake files with these flags for MSVC builds. Let me know if this breaks anything on the OS X build.

psxdev commented 8 years ago

build is fine with your changes.

I am tryng to run PSMoveConfigTool but it is asking for openovr path:

ERROR [OpenVRContext::startup] Failed to initialize OpenVR: Installation path could not be located (110)!
INFO [App::init] Failed to initialize OpenVR!
INFO [Renderer::init()] Initializing Renderer Context
ERROR [AssetManager::loadTexture] Failed to load: ./assets/textures/DK2diffuse.jpg()
ERROR [App::init] Failed to initialize asset manager!
ERROR [App::exec] Failed to initialize application!
logout

directly from xcode is fine.

HipsterSloth commented 8 years ago

The OpenVR error can be ignored. All that means is that there will be no HMD alignment step in the tracker calibration stage.

The asset issue however means that the \src\psmoveconfigtool\assets folder isn't getting copied into the same folder that PSMoveConfig tool is running from. I think we might be missing some of the INSTALL actions in the cmake scripts on osx that do that. But you can manually copy that folder and it should work. The asset folder is needed because it contains textures and fonts that the config tool uses.

HipsterSloth commented 8 years ago

@psxdev Sorry I haven't done any more work on this in the last few months. I ran into a wall with the iso transfers working in libusb and then had to move onto DualShock4 support.

I just saw this tweet: "Magic :P @jldiezr @alvaroprieto new ps4 camera plug and play"

By "plug and play" did you mean that the newer version of the PSVR camera not need any custom libusb driver work? Or that is just works with the existing PS4EyeCam driver?

psxdev commented 8 years ago

Same code in osx solderless with custom connector with new camera model. I was testing this summer usbdk instead that crappy zadig stuff and i got initialization run fine under Windows(virtualbox guest in my OS X ) i need to do more test with new camera but i believe that we are on the way

cr4zyw3ld3r commented 7 years ago

Can anyone please repost the link to the adapter from alieexpress, seems to go nowhere for me

psxdev commented 7 years ago

https://www.amazon.com/Power-Socket-Connector-Playstation-Sensor-Camera/dp/B012IIH02W

ShinyZeni commented 7 years ago

amazon link is showing the item currently unavailable.

https://www.aliexpress.com/item/For-Playstation-4-PS4-Move-Eye-Camera-Power-Charging-port-Connector-Charging-Socket/32515899256.html?

robsonswiss commented 7 years ago

Hi

Any update to getting this camera working on Windows?

mirh commented 7 years ago

I'm relatively sure we are being blocked by libusb devs being slow as fuck reviewing PRs https://github.com/stbnps/PS4-Eye-Driver/issues/2

robsonswiss commented 7 years ago

This guy got it working, but no explanation. https://www.youtube.com/watch?v=55OGD0rSnHI

DejayRezme commented 7 years ago

He has some info here on his other videos: http://blog.livedoor.jp/tmako123-programming/archives/50520932.html He also has an interesting video for "ORBSLAM2" on his channel which might or might not be done using the PS4 eye.

I always wondered why the PS4 eye isn't used more for enthusiast computer vision. I'll have to get it working one day. Thanks for your sharing your awesome work @psxdev!

HipsterSloth commented 7 years ago

@DejayRezme Thanks for the link to tmako123's blog. That guy has some really cool computer vision stuff on there.

I was able to get both the OpenCV test app using stbnps's version of PS4-Eye-Driver and then compile the test apps. Sadly I still seem to be stuck at the same issue I always have with my PS4Camera where the program stalls out trying to get a video frame from the camera:

If anyone wants to try and build this themself I wrapped up all the code outlined in that blog post into a single github project.

You can build this for yourself like so:

git clone --recursive https://github.com/HipsterSloth/PS4-Eye-Driver.git
Run InitialSetup_X64.bat for 64-bit or InitialSetup_Win32.bat for 32-bit
Open build\PS4EYECam.sln
Build the "opencv" project first
- This will download opencv and build it in the deps folder
Build the solution
Build the "INSTALL" project
- This will copy the compiled executables to the x64 or Win32 folder

Or you can download the prebuild x64 binaries I built here and give it a try:

PS4-Eye-Driver.zip

You need to run OPENCVTest.exe once first after you have assigned the libusbk driver to the "usbboot" device to the ps4camera in zadig to install the firmware. The second time you run OPENCVTest.exe it should open the camera. However, you'll likely just see the gray windows like me.

DejayRezme commented 7 years ago

Pardon my language, but HOLY SHIT IT WORKS!

I pulled out my PS4 Eye that I bought and modified like 2 years ago but never did anything because I am too lazy for the cpp build system. I downloaded libussk dev kit and then zandig and only did step 4 (Installed lib-usb onto the USB boot device) and started opencvtest first then EWCTest and it worked! The second time I started it though there was a crash though and my usb hub or mouse and keyboard froze until the usb system recovered. "Last USB device malfunctioned". After that I have to replug the PS4 eye. The picture is also very dark.

I'm not finding the device in the device manager though.

PS: This really is a quite amazing camera though. Thanks again for all your work guys!

HipsterSloth commented 7 years ago

@DejayRezme Huh. That's cool, but also surprising. I wonder if either: A) The PS4 camera adapter I made isn't working right (I swear it worked on linux) B) I have a model of PS4 camera that isn't compatible with this driver (assuming there is more than one model).

My camera's model number is: CUH-ZEY1:

I'd be curious what model your camera is.

As for the camera being dark,there are a bunch of camera control functions in ps4eye.h that could be exposed in the OpenCV example:

https://github.com/HipsterSloth/PS4-Eye-Driver/blob/master/src/ps4eye.h#L152-L165

In the EWC test I assume that you use EWC_SetValue() to adjust a camera setting:

https://github.com/HipsterSloth/PS4-Eye-Driver/blob/master/thirdparty/EWC/ewclib.h#L412

using one of these properties:

https://github.com/HipsterSloth/PS4-Eye-Driver/blob/master/thirdparty/EWC/ewclib.h#L226-L243

As for the crash I won't be able to dig into that until I can get the camera working on my end. But I'm glad to hear someone finally had some luck with this!

DejayRezme commented 7 years ago

I'm glad too :) Yeah I assumed there are exposure controls. At least with enough light this camera gives great pictures. Also I didn't compile this, might get into that later though. But the whole C++ build ecosystem and me don't like each other. For the wiring I soldered the connection according to PSXDEVs instructions. To be honest, I'm surprised it worked as well.

Top right: Model CUH-ZEY1 Bottom right: 4-475-456-02 Middle right(not left like yours) CAN ICES-3(B)/NMB-3(B)

Hope this helps. I've seen that the newest PS4 cameras have changed too and have a round body now.

CrossVR commented 7 years ago

I'm trying it out with the second version of the PS4 Camera:

ps4camera_1

Which has this information on the bottom:

Model CUH-ZEY2 4-595-150-01 CAN ICES-3(B)/NMB-3(B)

I also only get gray screens in OpenCV test app, but it seems likely that this new version will also require updated firmware.

For the connection I soldered the OSH Park PCB board with PS4 AUX connectors from aliexpress. I am able to connect the boot device and upload the firmware from the ps4eye repository. I used the libusbK drivers for the USB Boot, Video and Composite devices.

nicolaerosia commented 7 years ago

@psxdev "Cool. Congratulations. I am glad to see people using my code. I have full reversed Sony driver from 1.76 ps4 console kernel and i am working in a new firmware and driver code with same features than original. I need free time to finish it." You've extracted a new firmware from a Sony update? Do you mind sharing the firmware and code, even if it's not final? I would like to take a look, thanks!

nicolaerosia commented 7 years ago

@Armada651 have you got the new camera to work?

psxdev commented 7 years ago

@nicolaerosia you can dump firmware from following methods:

From ps4 kernel. Right now only firmware from 1.0x to 1.76 kernels are publically dumped.
If you have an usb3 sniffer you can dump camera firmware in all versions. Man in the middle approach

About new camera model it can run fine under OS X with code already released by me

psxdev commented 7 years ago

Old tweet https://twitter.com/psxdev/status/786954904690720768

nicolaerosia commented 7 years ago

@psxdev I have no knowledge of ps4 kernels. Is the firmware the same? I'll get my ps4 camera in the following few days and I will get to test, thanks, that's what I was worried about - if it's going to work or not.

psmoveservice / PSMoveService

Add support for the PS4 camera #35

[timestamp] [threadID] facility level [function call]