merge / skulls

pre-built coreboot images and documentation on how to flash them for Thinkpad Laptops
GNU General Public License v3.0
677 stars 65 forks source link

Weird issues on x230 after flashing #180

Closed joeyboon closed 3 years ago

joeyboon commented 3 years ago

Hi hopefully this is the right place to ask these questions :). I bought a Lenovo X230 as a Christmas holiday project, but ran into a really weird issue after successfully flashing both chips. First some details:

Model: Lenovo X230 Processor: Intel Core I7-3520M CPU @2.90Ghz SSD: Samsung SSD 750 EVO 500GB RAM 2X Kingston KCP316SD8/8

I used skulls x230-0.1.11 and Raspberry Pi with a clip to flash the top chip and neuter the Intel mangement engine. After that I flashed x230_coreboot_seabios_free. This was successful and after flashing I started with a clean install of Qubes. Before flashing I ran Qubes on the laptop with no issues whatsoever.

After flashing I noticed sdwdate in Whonix was unable to get the time in most cases. I also experienced issues when logging in different sites and apps. For example I'm not able to log in to Firefox Sync (it seems to time out as soon as I try to log in) or Signal for desktop (same issue, as soon as i scan the QR code it times out).

My first guess was a bug in Qubes. So I installed Debian and had the exact same issues. I'm able to surf the web, speed is not the issue, but I'm unable to log into the same sites/apps. I tried a wired connection to rule out the wireless (same issues), tried a different ISP (tethering via cellphone) (same issues). When I switched to Google Chrome instead of Firefox ESR the pages would sometimes load, but not always and this does not solve the issue for Signal for example.

After this I tried to reflash the bottom chip internally on Debian, which failed (see screenshot) Screenshot from 2020-12-27 15-28-11

I then flashed the bottom chip again with the clip, this time I went with skulls x230-0.1.11 x230_coreboot_seabios (so not the completely free version). I reinstalled Debian (just in case) and I still experienced the issues.

I've inlcuded the debuglog for Signal-Desktop Log Signal-Desktop.txt

I'm running out of ideas. Could be a time issue? Could be that some cryptographic function is not working properly? Could you please help me troubleshoot?

Thrilleratplay commented 3 years ago

@joeyboon For Signal, I would first check open bugs for that app. Skulls should have nothing to do with a user level app. As for the internal flashing, how are you internally flashing the bottom chip? There is no "top" and "bottom" chip when flashing internally, the system see's only the full 12mb.

joeyboon commented 3 years ago

@Thrilleratplay The app was working fine and was updated on Qubes just before flashing. I understand that skulls and the bios have nothing to do with a specific app, but since more sites and apps have the same problem I just mentioned them in the hope that you would be able to diagnose the problem (which is system specific (as in this laptop) and system wide (as in multiple apps, browsers and sites all have the same issue).

As for the internal flashing. I downloaded the latest release for the X230 (skulls x230-0.1.11) to the machine (running Debian 10) itself and installed flashrom. I updated /etc/default/grub to read GRUB_CMDLINE_LINUX_DEFAULT="iomem=relaxed quiet" and then updated by issuing the command sudo update-grub. I rebooted and then ran the script by sudo ./x230_skulls.sh. It asked me which flavor bios I wanted. I chose option 1.

I mentioned this because I thought it could be part of the bigger problem.

jcholsap commented 3 years ago

@joeyboon I'm willing to kick some ideas around with you.

  1. Ensure your programmer is communicating properly with the EEPROM. flashrom should auto recognize the chip. Also take three reads and compare that all three are the same.
  2. It does sound like a time issue. But your log file has the recent time. Perhaps the O/S is using NTP as it's source but the mobo clock batt is dead or low voltage? Or maybe in your diligence in using flashrom you forgot to reinstall the clock batt?
  3. It occurs to me that if you are using wired erthernet, then maybe the GBE firmware space could be corrupted, maybe? Just brainstorming.
  4. You rotated your Ponoma clip 180 degrees and reverse biased the EEPROM long enough to damage it? I've done that before and got strange behaviour after fixing the clip.
  5. A stray lead with voltage touched the mobo and reverse biased the wrong IC at just the wrong place and damaged the mobo? But it's really unlikely your programmer could do that. But I once saw an inexperienced tech set a motherboard on an antistatic pad and powered it on. -_- Later the customer ended up returning the laptop because it flaked out.
joeyboon commented 3 years ago

@jcholsap Thanks for helping me diagnose! :)

  1. Flashing externally went okay. As mentioned I even reflashed the bios to check if this was the problem.
  2. The CMOS battery cable indeed was damaged (it's a second hand computer), see picture below. IMG_20201227_190239 I replaced it immediately, but unfortunately this did not fix the issue.
  3. The problem occurs on both wifi and ethernet.
  4. I'm sure I placed the clip the right way round.
  5. I've been very careful, I powered down the programmer every time I needed to move the clip, but of course this could be the problem.

I was thinking I could reflash both chips with the backups and see if the problem still occurs to see if this is a hardware issue. Can i do this with the external flash script? Something like sudo ./external_install_bottom.sh -i <backupname> And sudo ./external_install_top.sh -i <backupname> ?

jcholsap commented 3 years ago
  1. I have experienced where the cable lengths were altogether too long - or Chinese cables too cheap - and could not get a consistent result without downgrading the bit rate to something like 2400 bps. But jumping my Ponoma clip directly onto RPI I/O pins fixed it. Only multiple reads and file checks indicated the problem. It was a long weekend... I ended up searching for and buying 100% copper leads.
  2. Looks like the palmrest screw met the battery lead. You're probably okay. But a couple of things to look for: Could the exposed cooper have contacted an IC on the circuit board anywhere? (That would have grounded it.) Is it possible the RTC was at 01/01/1970 00:00:00 during flash? (Who knows what/if that could have done anything.)

Recovery 101 best practice is to return to a known good state. So, yes, I'd restore the backup images. I've not used Tom's scripts -- yet (planning to soon though) -- but I think he has docs on how to restore. Also, I think his scripts automatically does the three copies and compares for integrity. (So my first point might be superfluous.)

joeyboon commented 3 years ago

@jcholsap Thanks for your fast reply. Apparently there is no scipt yet to restore the backups, I could only find #71. Since I'm rather new to this I would need a script like that ;). Any other suggestions why externally flashing is no problem, but flashing internally fails?

jcholsap commented 3 years ago

Well, the 1vyrain is a recent development (past year) and I haven't had time. So, I don't know. But external flashing is the fallback method and applies to your situation. BTW, many folk have had difficulties with 1vyrain as it has matured.

There's probably no script b/c it's actually not difficult. Just need to dig a little into the flashrom utility for the flags. Basically you are going to write the EEPROM image. See the man page but also there are plenty of exmaples out there. Be sure to verify after write.

joeyboon commented 3 years ago

Hi all,

After many hours of troubleshooting I finally found out it was in fact network issue that had nothing to do with flashing the bios... Sorry for bothering you with this. Thanks for the your quick replies and great work! I'll close the issue.